This paper deals with a natural stochastic optimization procedure derived from the so-called Heavy-ball method differential equation, which was introduced by Polyak in the 1960s with his seminal contribution [Pol64]. The Heavy-ball method is a second-order dynamics that was investigated to minimize convex functions f .…
In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…
New method shows stochastic momentum can converge quickly on optimization problems.
problem Improving convergence of stochastic optimization methods.
method Stochastic heavy ball momentum with minibatching.
result Stochastic heavy ball momentum retains fast linear rate on quadratic problems.
Heavy Ball method speeds up finding global optima in non-convex problems.
problem Finding global optima in non-convex optimization problems.
method Heavy Ball momentum in non-convex optimization.
result Heavy Ball helps iterates enter a benign region faster, containing a global optimal point.
In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a re…
The paper analyzes convergence rates for SGD and SHB methods.
problem Analyzing convergence rates for stochastic gradient descent and heavy ball methods.
method Stochastic gradient descent and stochastic heavy ball method for general stochastic approximation problems.
result The last iterate of SHB converges almost surely to a minimizer and has faster convergence rates than SGD.
A new Bayesian filtering method speeds up stochastic Newton optimization.
problem Minimizing log-convex functions using stochastic methods.
method Contextualizes the problem as Bayesian inference, applying Bayesian filtering to update estimates.
result Establishes conditions for diminishing effect of older observations, akin to momentum.
Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…
Paper analyzes SHB method for neural networks, proving stability, connectivity, and global convergence.
problem Theoretical understanding of SHB method for neural networks.
method Mean-field analysis of SHB dynamics related to a partial differential equation.
result SHB method converges to global optimum and exhibits stability and connectivity.
Paper proves SHB convergence with biased gradients and approximate step sizes.
problem Establishing convergence of SHB with biased gradients and approximate step sizes.
method Generalizes SHB convergence conditions for biased gradients, approximate step sizes, and block updating.
result Proves convergence of SHB with new conditions for biased gradients and approximate step sizes.
Optimizes privacy-preserving optimization for heavy-tailed data.
problem Privacy-preserving optimization with heavy-tailed gradients.
method Pure ε-differential privacy framework for Lipschitz extensions.
result Minimax optimal excess-risk rate for pure ε-DP heavy-tailed SCO.
Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of t…
In this paper we study several classes of stochastic optimization algorithms enriched with heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic dual subspace ascent. This is the first time momentum variants of several of these metho…
Improved SHB method for faster convergence on strongly-convex quadratics.
problem Understanding and improving the theoretical and practical advantages of SHB.
method Noise-adaptive multi-stage algorithm for SHB with accelerated convergence.
result SHB can achieve accelerated convergence with larger mini-batch sizes.
A method for estimating the median of gradients in stochastic optimization.
problem Robust gradient estimation in stochastic optimization for various applications.
method Stochastic Proximal Point Method for median gradient estimation.
result The proposed method can converge even under heavy-tailed, state-dependent noise.
Optimizes web page freshness with limited crawling frequencies.
problem Maximize local cache freshness given crawling frequency constraints.
method Three novel online estimation schemes for page change rates.
result Convergent algorithms for estimating page change rates.
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dyna…
This paper analyzes two Lie group momentum optimization algorithms and their convergence rates.
problem Optimizing functions on Lie groups using momentum-based dynamics.
method Investigates Lie Heavy-Ball and Lie NAG-SC algorithms, quantifying their convergence rates under smoothness and convexity assumptions.
result Lie NAG-SC accelerates optimization over the momentumless case, while Lie Heavy-Ball does not.
There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …
The paper analyzes how momentum affects convergence in stochastic gradient methods.
problem Lack of clear understanding of momentum's impact on convergence and performance.
method Unified analysis of several popular algorithms using the QHM formulation.
result Provides practical guidelines for setting learning rate and momentum parameters.
Stochastic gradient methods can converge in expectation under heavy-tailed noise.
problem Convergence of stochastic gradient methods under heavy-tailed noise.
method Comprehensive study of stochastic optimization under heavy-tailed noise for e x t s f S G D extsf{SGD} e x t s f S G D , e x t s f S M D extsf{SMD} e x t s f S M D , e x t s f A S M D extsf{ASMD} e x t s f A S M D , e x t s f S G D M extsf{SGDM} e x t s f S G D M in convex and nonconvex optimization. result Established in-expectation convergence results for various stochastic gradient methods.
Optimized method tackles convex optimization with heavy-tailed noise.
problem Convex optimization problems with noisy gradients.
method Vanilla stochastic proximal subgradient method without gradient clipping or normalization.
result Achieves optimal complexity for various convex optimization types under heavy-tailed noise.
New class of heavy-tailed distributions shows weighted averages dominate individual variables.
problem Understanding and comparing risks in heavy-tailed distributions.
method Introducing a new class of heavy-tailed distributions and proving stochastic dominance relations.
result Weighted averages of random variables in this class are stochastically larger than individual variables.
Paper tackles DP-SCO with heavy-tailed data in high dimensions, improving error bounds.
problem Differentially private stochastic optimization with heavy-tailed data in high-dimensional spaces.
method Proposes methods for DP-SCO with polytope constraints and LASSO, analyzing sparsity constraints.
result Achieved near optimal error bounds for DP-SCO with heavy-tailed data.
Heavy-tailed distributions emerge in SGD's parameter evolution.
problem Understanding heavy-tailed distributions in SGD parameter evolution.
method Continuous diffusion approximation of SGD (homogenized SGD) analysis.
result Explicit upper and lower bounds on tail-index of homogenized SGD.
Paper examines the structure of stochastic gradients in deep learning.
problem Exploring the structure and heavy tails of stochastic gradients in deep learning.
method Conducted formal statistical tests on stochastic gradients and gradient noise.
result Stochastic gradients and gradient noise do not exhibit power-law heavy tails, but their covariance spectra do.
Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochast…
New algorithm improves heavy-tailed statistical estimation in streaming data.
problem Heavy-tailed statistical estimation in streaming data.
method Clipped stochastic gradient descent algorithm with improved analysis.
result Guarantees exponential concentration with O ( 1 ) O(1) O ( 1 ) batch size for mean estimation and linear regression. Develops a computationally tractable differentially private mean estimator called the balloon mean.
problem Robust mean estimation in the presence of outliers and heavy-tailed distributions.
method Iterative clipping procedure over Mahalanobis balls.
result Balloon mean is robust to outliers and outperforms existing estimators in contaminated settings.
New method bounds stochastic subgradient methods with heavy-tailed noise.
problem Bounding stochastic subgradient methods under heavy-tailed noise.
method Clipped version of projected stochastic subgradient method.
result Near optimal any-time and finite horizon bounds for averaging schemes.
Researchers study heavy-tail properties of SGD using stochastic recurrence equations.
problem Analyzing heavy-tail properties of Stochastic Gradient Descent (SGD).
method Modeling SGD iterations as multivariate affine stochastic recursions and applying the theory of irreducible-proximal (i-p) matrices.
result Extended results of Gürbüzbalaban et al. (2020) by using the theory of i-p matrices.
Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Po…
Studied how heavy-tailed behavior affects SGD's generalization in quadratic optimization.
problem Link between heavy-tailed behavior and generalization in SGD.
method Used heavy-tailed stochastic differential equation and proved stability bounds.
result Stability of SGD depends on the loss function's tail behavior.
Paper studies stochastic optimization methods with momentum, proving convergence and avoiding traps.
problem Optimizing non-convex functions with momentum.
method Unified analysis of stochastic gradient descent variants, including S-NAG and Adam.
result Convergence to critical points and avoidance of undesired critical points like local maxima or saddle points.
Stochastic optimization's success linked to heavy-tailed noise.
problem Understanding stochastic optimization's success mechanisms.
method Modeling stochastic optimization as random recurrence relations, analyzing multiplicative noise and heavy-tailed behavior.
result Multiplicative noise leads to heavy-tailed stationary behavior in optimization parameters.
Improved regret bounds for linear bandits with heavy-tailed rewards.
problem Stochastic linear bandits with heavy-tailed rewards.
method Elimination-based algorithm guided by experimental design.
result Regret bound of \(\tilde{\mathcal{O}}(d^\frac{1+3ε}{2(1+ε)} T^\frac{1}{1+ε})\) for \(ε\in (0,1)\).
Improved privacy-preserving methods for convex optimization with heavy-tailed data.
problem Privacy-preserving optimization of convex functions with heavy-tailed data.
method Developed algorithms for private mean estimation and convex optimization under concentrated differential privacy constraints.
result Achieved improved upper bounds on excess population risk for convex and strongly convex loss functions.
New UCB-type algorithms reduce regret bounds for stochastic bandits with heavy and super heavy noise.
problem Improving regret bounds for stochastic bandits with heavy-tailed noise.
method General convex optimization methods with an inexact oracle, Clipped-SGD-UCB algorithm.
result Achieved an O ( log T K T log T ) O(\log T\sqrt{KT\log T}) O ( log T K T log T ) regret bound for symmetric noise, better than general lower bounds. A new algorithm reduces regret in cooperative multi-agent bandits with heavy-tailed data.
problem Cooperative multi-agent bandits with heavy-tailed data.
method MP-UCB algorithm incorporating robust estimation with message-passing protocol.
result Optimal regret bounds for MP-UCB in various settings.
New bounds for heavy-tailed SDEs without info-theory terms.
problem Understanding generalization of heavy-tailed stochastic optimization.
method Fractional Fokker-Planck equation to estimate entropy flows.
result High-probability bounds with better dimension dependence.
Analysis of momentum methods on quadratic models, showing SGD's superiority.
problem Analysis of stochastic gradient algorithms with momentum on quadratic models.
method Inspired by random matrix theory, exact characterization of loss values.
result Stochastic heavy-ball momentum does not improve over SGD in the strongly convex setting.
New bounds link SGD's generalization to heavy tails without topological assumptions.
problem Linking SGD's generalization error to heavy tails without additional assumptions.
method Developed Wasserstein stability bounds for heavy-tailed SDEs and their discretizations, converting to generalization bounds.
result Generalization bounds for a broader class of objective functions, including non-convex functions, without topological assumptions.
Diversification improves profits for heavy-tailed investments.
problem Investment portfolios of Pareto-distributed returns.
method Stochastic dominance and majorization order.
result Diversification increases first-order stochastic dominance for heavy-tailed returns.
Study on SA with heavy-tailed and LRD noise, establishing finite-time bounds.
problem Analyzing stochastic approximation under heavy-tailed and LRD noise.
method Noise-averaging argument to regularize impact of non-classical noise.
result Established first finite-time moment bounds for SA under heavy-tailed and LRD noise.
Stochastic Volatility in Mean models with heavy-tailed distributions using Hidden Markov Models
problem Accurate inference for Stochastic Volatility in Mean models with heavy-tailed distributions
method Numerically stable estimation procedure and parallel computing
result Significant reduction in computational times
INF-clip optimizes heavy-tailed MAB problems with improved performance.
problem Optimizing multi-armed bandit problems with heavy-tailed rewards.
method INF-clip algorithm for adversarial and stochastic heavy-tailed MAB settings.
result INF-clip is optimal for linear and non-linear heavy-tailed stochastic MAB problems.
New methods solve optimization problems with heavy-tailed noise, improving upon existing complexity bounds.
problem Optimization problems with heavy-tailed noise and weakly average smoothness.
method Normalized stochastic first-order methods with Polyak, multi-extrapolated, and recursive momentum.
result First-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise.
Work on SGDm under heavy-tailed noise, revealing its generalization properties.
problem Understanding generalization of SGDm under heavy-tailed noise.
method Analysis of continuous-time limit (SDE) and discrete-time SGDm, establishing generalization bounds.
result SGDm can have worse generalization in the presence of heavy-tailed noise for quadratic loss functions.