arXiv research

Heavy Ball method speeds up finding global optima in non-convex problems.

Proposes momentum methods for Lie groups, improving on classical algorithms.

This paper deals with a natural stochastic optimization procedure derived from the so-called Heavy-ball method differential equation, which was introduced by Polyak in the 1960s with his seminal contribution [Pol64]. The Heavy-ball method is a second-order dynamics that was investigated to minimize convex functions f .…

Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dyna…

Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Po…

A new Bayesian filtering method speeds up stochastic Newton optimization.

Two new differentially private optimization algorithms derived from accelerated methods.

Paper analyzes SHB method for neural networks, proving stability, connectivity, and global convergence.

The paper analyzes convergence rates for SGD and SHB methods.

New step-size methods improve SHB convergence for stochastic optimization.

We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme.…

We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces. Our perspective leads to a generic and unifying non…

Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…

Paper proves SHB convergence with biased gradients and approximate step sizes.

Fine-grained analysis of gradient descent with momentum provides modified loss equations.

Study accelerates gradient methods in machine learning, revealing risk and stability connections.

In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…

New convergence rates for SGD under heavy-tailed noise with infinite variance.

New method shows stochastic momentum can converge quickly on optimization problems.

Derives new optimization methods using variational integrators.

In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a re…

New methods solve optimization problems with heavy-tailed noise, improving upon existing complexity bounds.

Two major momentum-based techniques that have achieved tremendous success in optimization are Polyak's heavy ball method and Nesterov's accelerated gradient. A crucial step in all momentum-based methods is the choice of the momentum parameter $m$ which is always suggested to be set to less than $1$. Although the choice…

Momentum SGD fails to track nonstationary optima due to drift amplification.

This paper analyzes two Lie group momentum optimization algorithms and their convergence rates.

Develops a computationally tractable differentially private mean estimator called the balloon mean.

Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stoch…

We obtain sharp bounds on the performance of Empirical Risk Minimization performed in a convex class and with respect to the squared loss, without assuming that class members and the target are bounded functions or have rapidly decaying tails. Rather than resorting to a concentration-based argument, the method used her…

Improved convergence for Polyak steps with momentum in smooth convex optimization.

Study accelerates optimization methods in non-convex problems, but doesn't improve the algorithm's performance.

Improved Sparse Polyak for high-dimensional M-estimation with sparser solutions.

Optimizes privacy-preserving optimization for heavy-tailed data.

Accelerated gradient (AG) methods are breakthroughs in convex optimization, improving the convergence rate of the gradient descent method for optimization with smooth functions. However, the analysis of AG methods for non-convex optimization is still limited. It remains an open question whether AG methods from convex o…

Formulae for Vassiliev invariants derived from Kauffman polynomial.

Sparse Polyak improves high-dimensional statistical estimation.

Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the rol…

Optimizes web page freshness with limited crawling frequencies.

Improved SHB method for faster convergence on strongly-convex quadratics.

A novel decentralized deep learning algorithm using gradient-based optimization.

Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning

The paper improves convergence for linear systems using entropic mirror descent with Polyak stepsizes.

Proposes a new adaptive learning rate for SGD.

Polyak's momentum accelerates training of neural networks.

Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of t…

Polyak step size GD reaches final radius of convergence after log iterations.

Optimization algorithms help overparameterized neural networks achieve high performance.

Differentially quantized gradient methods improve convergence in noisy communication channels.

In this paper we study several classes of stochastic optimization algorithms enriched with heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic dual subspace ascent. This is the first time momentum variants of several of these metho…