arXiv research

This paper deals with a natural stochastic optimization procedure derived from the so-called Heavy-ball method differential equation, which was introduced by Polyak in the 1960s with his seminal contribution [Pol64]. The Heavy-ball method is a second-order dynamics that was investigated to minimize convex functions f .…

In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…

New method shows stochastic momentum can converge quickly on optimization problems.

Heavy Ball method speeds up finding global optima in non-convex problems.

In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a re…

The paper analyzes convergence rates for SGD and SHB methods.

A new Bayesian filtering method speeds up stochastic Newton optimization.

Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…

Paper analyzes SHB method for neural networks, proving stability, connectivity, and global convergence.

Paper proves SHB convergence with biased gradients and approximate step sizes.

Optimizes privacy-preserving optimization for heavy-tailed data.

Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of t…

In this paper we study several classes of stochastic optimization algorithms enriched with heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic dual subspace ascent. This is the first time momentum variants of several of these metho…

Improved SHB method for faster convergence on strongly-convex quadratics.

A method for estimating the median of gradients in stochastic optimization.

Optimizes web page freshness with limited crawling frequencies.

Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dyna…

This paper analyzes two Lie group momentum optimization algorithms and their convergence rates.

There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …

Stochastic gradient methods can converge in expectation under heavy-tailed noise.

Optimized method tackles convex optimization with heavy-tailed noise.

New class of heavy-tailed distributions shows weighted averages dominate individual variables.

The use of momentum in stochastic gradient methods has become a widespread practice in machine learning. Different variants of momentum, including heavy-ball momentum, Nesterov's accelerated gradient (NAG), and quasi-hyperbolic momentum (QHM), have demonstrated success on various tasks. Despite these empirical successe…

Paper tackles DP-SCO with heavy-tailed data in high dimensions, improving error bounds.

Heavy-tailed distributions emerge in SGD's parameter evolution.

Paper examines the structure of stochastic gradients in deep learning.

Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochast…

New algorithm improves heavy-tailed statistical estimation in streaming data.

Develops a computationally tractable differentially private mean estimator called the balloon mean.

New method bounds stochastic subgradient methods with heavy-tailed noise.

Researchers study heavy-tail properties of SGD using stochastic recurrence equations.

Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Po…

Studied how heavy-tailed behavior affects SGD's generalization in quadratic optimization.

Stochastic optimization's success linked to heavy-tailed noise.

Paper studies stochastic optimization methods with momentum, proving convergence and avoiding traps.

Improved regret bounds for linear bandits with heavy-tailed rewards.

Improved privacy-preserving methods for convex optimization with heavy-tailed data.

New UCB-type algorithms reduce regret bounds for stochastic bandits with heavy and super heavy noise.

A new algorithm reduces regret in cooperative multi-agent bandits with heavy-tailed data.

New bounds for heavy-tailed SDEs without info-theory terms.

New bounds link SGD's generalization to heavy tails without topological assumptions.

Analysis of momentum methods on quadratic models, showing SGD's superiority.

Diversification improves profits for heavy-tailed investments.

Study on SA with heavy-tailed and LRD noise, establishing finite-time bounds.

Stochastic Volatility in Mean models with heavy-tailed distributions using Hidden Markov Models

INF-clip optimizes heavy-tailed MAB problems with improved performance.

New methods solve optimization problems with heavy-tailed noise, improving upon existing complexity bounds.

Work on SGDm under heavy-tailed noise, revealing its generalization properties.