Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
Improved SGD methods converge faster for nonconvex optimization.
problem Nonconvex optimization challenges in machine learning.
method Adaptive SGD with line-search and Polyak stepsizes.
result Unified convergence rates for various nonconvex functions.
Adaptive-SGD method optimizes machine learning training with dynamic batch and step sizes.
problem Optimizing machine learning training with adaptive batch and step sizes.
method Adaptive-SGD method that dynamically adjusts batch size and step size based on local curvature and probability of descent directions.
result Adaptive-SGD achieves global linear convergence on self-concordant functions and compares favorably to fine-tuned methods.
Introduces a new stochastic optimization method for deep learning.
problem Minimizing loss functions in deep neural networks.
method Introduces a second-order stochastic Runge-Kutta method and an adaptive SGD-G2.
result The method yields consistent minimization of loss functions and automatically adjusts learning rates.
Paper analyzes high probability convergence of adaptive SGD with momentum.
problem Theoretical understanding of adaptive SGD with momentum in nonconvex settings is incomplete.
method High probability analysis under weak assumptions.
result First high probability convergence proof for gradients to zero in Delayed AdaGrad with momentum.
AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.
problem Optimal convergence rates for non-convex, smooth objectives with adaptive step sizes.
method Adaptive SGD (AdaGrad-Norm) with self-tuning step sizes, analyzing under unbounded gradients and affine variance scaling.
result AdaGrad-Norm achieves order optimal convergence rate of $\mathcal{O}\left(\frac{\mathrm{poly}\log(T)}{\sqrt{T}}
ight)$ under optimal assumptions.
New adaptive SGD algorithms for federated learning over physical channels.
problem Reducing communication cost in federated learning over physical channels.
method Proposed adaptive federated SGD algorithms considering channel noise and hardware constraints.
result Demonstrated convergence rates adaptive to stochastic gradient noise level.
Adam optimizer converges to zeros of a new vector field, not just gradient zeros.
problem Prove convergence rates for Adam optimizer in simple quadratic optimization problems.
method Introduced Adam vector field to analyze Adam optimizer's convergence.
result Established optimal convergence rates for Adam optimizer.
Adaptive SGD learns optimal batch size for strong convex functions.
problem Finding optimal batch size for SGD in practice.
method Adaptive SGD method that learns optimal batch size.
result Adaptive SGD exhibits nearly optimal performance in experiments.
Stochastic convex optimization algorithms are the most popular way to train machine learning models on large-scale data. Scaling up the training process of these models is crucial, but the most popular algorithm, Stochastic Gradient Descent (SGD), is a serial method that is surprisingly hard to parallelize. In this pap…
Private adaptive methods improve on traditional SGD for convex optimization.
problem Differential privacy constraints in gradient optimization.
method Differentially private variants of SGD and AdaGrad with adaptive stepsizes and non-isotropic clipping.
result Private AdaGrad outperforms private SGD in high-dimensional problems.
AdaS adapts SGD learning rate based on knowledge gain metrics.
problem Empirical step-size selection in SGD optimization lacks consistency and insight.
method Introduces AdaS algorithm that adapts SGD learning rate based on knowledge gain metrics.
result AdaS outperforms existing adaptive learning methods in convergence and generalization.
Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.
problem Analyzing risk and learning rate dynamics in high-dimensional optimization problems.
method Developed a framework to give exact expressions for risk and learning rate curves using ODEs.
result Exact expressions for risk and learning rate curves, with detailed analysis of two adaptive learning rates.
In stochastic gradient descent, especially for neural network training, there are currently dominating first order methods: not modeling local distance to minimum. This information required for optimal step size is provided by second order methods, however, they have many difficulties, starting with full Hessian having…
Adapts SGD to noise and problem specifics for faster convergence.
problem Minimizing smooth, strongly-convex functions with varying noise and problem constants.
method Adaptive SGD with exponentially decreasing step-sizes, Nesterov acceleration, and stochastic line-search.
result Achieves near-optimal convergence rates without knowing noise or problem specifics.
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
New algorithms improve SGD convergence and reduce variance for over-parameterized models.
problem Slower convergence in non-interpolation settings for SGD variants.
method Proposed AdaSPS and AdaSLS with variance reduction for robust convergence.
result Achieves faster convergence rates and robustness in non-interpolation settings.
SGD methods fail to converge to global minimizers in deep neural networks with ReLU activation.
problem Failure of SGD methods to converge to global minimizers in deep neural networks.
method Stochastic Gradient Descent (SGD) and its variants like Adam, RMSProp, etc.
result SGD methods fail to converge to global minimizers with high probability in deep neural networks with ReLU activation.
New method for faster convergence in non-convex optimization with unbounded smoothness.
problem Finding first-order stationary points of non-convex functions with unbounded smoothness.
method Developed a stopped analysis technique to prove convergence rates for ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smooth functions. result Achieved O ( p o l y log ( T ) T ) \mathcal{O}(\frac{\mathrm{poly}\log(T)}{\sqrt{T}}) O ( T poly l o g ( T ) ) convergence rates without uniform noise bounds.