arXiv research

This paper explains why Adam generalizes worse than SGD by analyzing its components.

Derives effective continuous dynamics for adaptive SGD methods.

Untuned SGD converges but with an exponential dependence on smoothness, adaptive methods prevent this.

Adaptive step-size method improves compressed SGD performance in machine learning.

This paper analyzes convergence of DP-SGD with adaptive quantile clipping.

New insights into SGD and SGD-M in high dimensions.

Adaptive distributed SGD reduces delay in slow workers.

Introduces a new stochastic optimization method for deep learning.

To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple workers is limited by the communication overhead. Two approaches, namely pipelining and gradient spar…

Paper analyzes high probability convergence of adaptive SGD with momentum.

Adaptive quantization improves SGD accuracy in data-parallel settings.

AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.

From a simplified analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned. We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search. I…

Private adaptive methods improve on traditional SGD for convex optimization.

New adaptive SGD algorithms for federated learning over physical channels.

Adaptive SGD learns optimal batch size for strong convex functions.

IntSGD compresses SGD gradients without floats, converging as SGD.

Adapts SGD to noise and problem specifics for faster convergence.

New method trains deep networks robustly without adaptive methods.

Proposes a new adaptive learning rate for SGD.

AdaSGD combines SGD and Adam benefits, eliminating the need for transition.

ARFF reduces spectral bias in SGD-trained neural networks.

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drasti…

Improved SGD methods converge faster for nonconvex optimization.

Develops a parameter-free SGD algorithm with optimal convergence rate.

Improves distributed SGD convergence speed with reduced computation load.

End-to-end analysis of SGD for STL with adaptive sub-sampling.

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the gene…

Adaptive-SGD method optimizes machine learning training with dynamic batch and step sizes.

Stochastic Gradient Descent (SGD) is one of the most widely used techniques for online optimization in machine learning. In this work, we accelerate SGD by adaptively learning how to sample the most useful training examples at each time step. First, we show that SGD can be used to learn the best possible sampling distr…

Unified analysis for decentralized SGD across various topologies and updates.

Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…

This research explains why SGD generalizes better than ADAM in deep learning.

This paper analyzes SGD with biased gradients for deep learning models.

Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradie…

AdaScale SGD adapts learning rates for large-batch training efficiently.

Improved analysis shows momentum in SGD reduces batch size needs for non-convex objectives.

AdaS adapts SGD learning rate based on knowledge gain metrics.

Stochastic Gradient Langevin Dynamics infuses isotropic gradient noise to SGD to help navigate pathological curvature in the loss landscape for deep networks. Isotropic nature of the noise leads to poor scaling, and adaptive methods based on higher order curvature information such as Fisher Scoring have been proposed t…

Improved loss scaling for stochastic momentum algorithms in high dimensions.

CSER improves SGD efficiency by resetting errors and partial synchronization.

AdaGrad outperforms SGD in non-convex optimization problems by a factor of d.

AdaBelief optimizes deep learning models with faster convergence and better stability.

GALA adapts learning rates online by aligning gradients, improving deep learning model performance.

Although stochastic gradient descent (SGD) method and its variants (e.g., stochastic momentum methods, AdaGrad) are the choice of algorithms for solving non-convex problems (especially deep learning), there still remain big gaps between the theory and the practice with many questions unresolved. For example, there is s…

Optimization algorithms affect fairness in deep learning models, especially with adaptive methods like RMSProp.

The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.

We propose an algorithm for the adaptation of the learning rate for stochastic gradient descent (SGD) that avoids the need for validation set use. The idea for the adaptiveness comes from the technique of extrapolation: to get an estimate for the error against the gradient flow which underlies SGD, we compare the resul…