arXiv research

Paper proves multiplicative weight updates can train neural networks without learning rate tuning.

The (stochastic) gradient descent and the multiplicative update method are probably the most popular algorithms in machine learning. We introduce and study a new regularization which provides a unification of the additive and multiplicative updates. This regularization is derived from an hyperbolic analogue of the entr…

In this letter, we generalize the convolutional NMF by taking the $β$-divergence as the contrast function and present the correct multiplicative updates for its factors in closed form. The new updates unify the $β$-NMF and the convolutional NMF. We state why almost all of the existing updates are inexact and approximat…

In this paper, we extend the $β$-CNMF to two dimensions and derive exact multiplicative updates for its factors. The new updates generalize and correct the nonnegative matrix factor deconvolution previously proposed by Schmidt and Mørup. We show by simulation that the updates lead to a monotonically decreasing $β$-dive…

New algorithm for online optimization over symmetric cones, unifying previous methods.

LD-SGD improves communication in decentralized SGD.

Online learning makes sequence of decisions with partial data arrival where next movement of data is unknown. In this paper, we have presented a new technique as multiple times weight updating that update the weight iteratively forsame instance. The proposed technique analyzed with popular state-of-art algorithms from …

The paper introduces MU for NMF with $β$-divergences and disjoint constraints.

Multiple-step lookahead policies have demonstrated high empirical competence in Reinforcement Learning, via the use of Monte Carlo Tree Search or Model Predictive Control. In a recent work \cite{efroni2018beyond}, multiple-step greedy policies and their use in vanilla Policy Iteration algorithms were proposed and analy…

An algorithm for computing positive semidefinite factorizations of matrices.

Identifying recurring patterns in high-dimensional time series data is an important problem in many scientific domains. A popular model to achieve this is convolutive nonnegative matrix factorization (CNMF), which extends classic nonnegative matrix factorization (NMF) to extract short-lived temporal motifs from a long …

Federated learning can be vulnerable to adversarial attacks, which this work addresses.

New algorithm speeds up NMF with $β$-divergence.

Nonnegative matrix factorization (NMF), a dimensionality reduction and factor analysis method, is a special case in which factor matrices have low-rank nonnegative constraints. Considering the stochastic learning in NMF, we specifically address the multiplicative update (MU) rule, which is the most popular, but which h…

Regression problems that have closed-form solutions are well understood and can be easily implemented when the dataset is small enough to be all loaded into the RAM. Challenges arise when data is too big to be stored in RAM to compute the closed form solutions. Many techniques were proposed to overcome or alleviate the…

We shed new insights on the two commonly used updates for the online $k$-PCA problem, namely, Krasulina's and Oja's updates. We show that Krasulina's update corresponds to a projected gradient descent step on the Stiefel manifold of the orthonormal $k$-frames, while Oja's update amounts to a gradient descent step using…

SlowMo improves distributed SGD by periodically synchronizing and using momentum, leading to better accuracy.

NNEinFact fits any nonnegative tensor factorization quickly and accurately.

New framework provides privacy guarantees for practical federated learning.

NMF with specific constraints is equivalent to LDA.

The SCMU algorithm computes cone factorizations for symmetric cones, improving upon existing methods.

New algorithm provably converges to second-order stationary points in NMF.

In this paper, we provide a novel construction of the linear-sized spectral sparsifiers of Batson, Spielman and Srivastava [BSS14]. While previous constructions required $Ω(n^4)$ running time [BSS14, Zou12], our sparsification routine can be implemented in almost-quadratic running time $O(n^{2+\varepsilon})$. The funda…

Synthetic data replaces model updates in federated learning.

Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…

Improved ADMM for convex distributed learning with differential privacy.

The thesis clarifies when local updates outperform centralized methods in heterogeneous data environments.

Flexible framework for optimal trading across multiple asset venues.

Framework for safely updating machine learning models.

FedLoRU improves FL efficiency by using low-rank updates.

A method learns common bias for multiple low-variance tasks without hyper-parameter tuning.

The multiplicative update (MU) algorithm has been extensively used to estimate the basis and coefficient matrices in nonnegative matrix factorization (NMF) problems under a wide range of divergences and regularizers. However, theoretical convergence guarantees have only been derived for a few special divergences withou…

Although reinforcement learning has made great strides recently, a continuing limitation is that it requires an extremely high number of interactions with the environment. In this paper, we explore the effectiveness of reusing experience from the experience replay buffer in the Deep Q-Learning algorithm. We test the ef…

Proposes RNSE for clustering with adaptive similarity matrix learning.

New hierarchical tensor decomposition model for complex data.

FedCluster accelerates federated learning convergence by cycling device groups.

Study on incentivizing truthfulness in federated learning with heterogeneous data.

The stochastic gradient (SG) method can minimize an objective function composed of a large number of differentiable functions, or solve a stochastic optimization problem, to a moderate accuracy. The block coordinate descent/update (BCD) method, on the other hand, handles problems with multiple blocks of variables by up…

We propose a framework based on distributional reinforcement learning and recent attempts to combine Bayesian parameter updates with deep reinforcement learning. We show that our proposed framework conceptually unifies multiple previous methods in exploration. We also derive a practical algorithm that achieves efficien…

BI-MAML learns multiple tasks without forgetting old ones.

We study a stochastic multiplicative system composed of finite asynchronous elements to describe the wealth evolution in financial markets. We find that the wealth fluctuations or returns of this system can be described by a walk with correlated step sizes obeying truncated Levy-like distribution, and the cross-correla…

Proposes a method to balance tasks in multitask learning with a single gradient step update.

Algorithm learns Sherrington-Kirkpatrick model parameters at low temperatures.

This work analyzes how users and services adapt to reduce risk, leading to specialization.

MetaGrad adapts multiple learning rates for faster online optimization.

FedSGM tackles constrained federated learning with unified framework.

Federated learning improves by unbiased gradient aggregation and controllable meta updating.

A new FL algorithm reduces communication overhead by selectively updating model parameters.