Hill-ADAM optimizes loss landscapes by exploring state space deterministically.
problem Escaping local minima in loss landscapes.
method Hill-ADAM alternates between minimizing and maximizing error to explore the loss space.
result Hill-ADAM finds the global minimum state in loss landscapes.
APMSqueeze improves Adam for faster training with less communication.
problem Adam's inefficiency in parallel training due to gradient compression limitations.
method Error-compensated gradient compression for Adam-preconditioned Momentum SGD.
result Significantly reduces communication time per epoch without sacrificing accuracy.
AdaX improves Adam by exponentially accumulating past gradients, leading to better performance in machine learning tasks.
problem Adam's fast convergence can lead to local minimums in non-convex problems.
method AdaX exponentially accumulates past gradients to adaptively tune the learning rate.
result AdaX outperforms Adam in various machine learning tasks, including computer vision and natural language processing.
New algorithms improve generalization of Adam and AdamW optimizers.
problem Adam and AdamW optimizers generalize worse than SGD.
method Algorithmic stability analysis and clever momentum-based SGD integration.
result HomeAdam(W) algorithms achieve better generalization and faster convergence.
First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of ``long-term memory" in Adam-like algorithms, which could hamper their performance and lead to divergence. In our s…
Adam and similar methods fail to converge in some deep learning tasks.
problem Adam and similar methods fail to converge to optimal solutions in some deep learning tasks.
method Analysis of gradient updates scaled by square roots of exponential moving averages of squared past gradients. Proposed new variants of Adam with `long-term memory'.
result New variants of Adam algorithms fix convergence issues and often lead to improved empirical performance.
Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this wo…
Untuned warmup of Adam performs similarly to RAdam in practical settings.
problem The stability of adaptive optimization algorithms like Adam requires a warmup schedule.
method Provided alternative explanation for warmup necessity based on update term magnitude.
result Untuned warmup of Adam is as effective as RAdam in typical practical settings.
Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.
problem Inefficiencies in ad-hoc learning rate tuning for optimization algorithms.
method Developed AdaGrad++ and Adam++ without predefined learning rates and proved their convergence.
result AdaGrad++ and Adam++ achieve comparable convergence rates to AdaGrad and Adam respectively.
Adam converges to stationary points under relaxed conditions.
problem Understanding and proving convergence of Adam under realistic assumptions.
method New proof of boundedness of gradients and variance-reduced Adam.
result Adam converges to ε-stationary points with O(ε⁻⁴) gradient complexity under realistic conditions.
Paper analyzes convergence of Adam-type RL algorithms under Markovian sampling.
problem Theoretical convergence analysis of Adam-type RL algorithms.
method Develops techniques for analyzing convergence under Markovian sampling.
result PG-AMSGrad and TD-AMSGrad converge to stationary points or global optima at specified rates.
The paper analyzes Adam-type algorithms for non-convex optimization and proves their convergence rate.
problem The convergence of Adam-type algorithms for non-convex optimization problems.
method The paper studies a class of adaptive gradient-based momentum algorithms that update search directions and learning rates simultaneously.
result The Adam-type methods can achieve a convergence rate of order O ( log T / T ) O(\log{T}/\sqrt{T}) O ( log T / T ) for nonconvex stochastic optimization. The study examines the dynamic behavior of RMSprop and Adam algorithms.
problem Understanding the training loss curve patterns of adaptive gradient algorithms.
method Careful numerical experiments and theoretical explanations using the signGD flow.
result Adam converges smoother and faster when momentum factors are close to each other.
New convergence analysis for ADAM algorithm in non-convex optimization with adaptive step size.
problem Convergence issues in ADAM algorithm for non-convex optimization.
method Study of ADAM algorithm under bounded adaptive step size assumption, providing safe step sizes.
result Novel first order convergence rate result in deterministic and stochastic contexts.
This research explains why SGD generalizes better than ADAM in deep learning.
problem Understanding the generalization gap between SGD and ADAM in deep learning.
method Analyzing local convergence behaviors through Levy-driven stochastic differential equations (SDEs).
result SGD is more locally unstable and better escapes from sharp minima to flatter ones, leading to better generalization.
Adam optimizes non-convex functions, converging to critical points.
problem Finding local minima of non-convex functions.
method Continuous-time model of Adam, ODE approximation, decreasing stepsize.
result Adam converges to critical points of non-convex functions.
Paper studies Adam's convergence under relaxed assumptions, proving a rate of O(poly(log T)/sqrt(T)).
problem Understanding Adam's convergence in non-convex, stochastic optimization with unbounded gradients and noise.
method Introduced a comprehensive noise model and used it to prove Adam's convergence rate.
result Adam finds a stationary point with a rate of O(poly(log T)/sqrt(T)) in high probability.
This paper improves Adam's performance in machine learning tasks.
problem Improving the generalization ability of adaptive gradient methods.
method Develops a control theoretic framework to propose AdamSSM, a new variant of Adam.
result AdamSSM improves generalization accuracy and convergence compared to recent adaptive gradient methods.
Enhanced Adam uses higher-order moments for better performance.
problem Improving the performance of Adam optimization algorithm.
method Proposes HAdam, an extension of Adam using higher-order moments of the stochastic gradient.
result Higher-order moments of the stochastic gradient can lead to better performance than vanilla Adam.
A new sufficient condition ensures convergence of Adam and RMSProp in non-convex settings.
problem Adam and RMSProp diverge in convex settings, despite being influential for deep learning.
method Introduces a sufficient condition based on base learning rate and historical second-order moments.
result Guarantees global convergence of generic Adam/RMSProp in non-convex optimization.
Adam optimizer fails to stay close to optimal point under certain conditions.
problem Adam optimizer's tendency to deviate from the optimal point in training neural networks.
method Analyzed Adam's behavior in convex regions and proposed a new algorithm to correct this.
result Adam optimizer cannot stay close to the optimal point when effective learning rate exceeds a certain bound.
EAdam optimizer improves performance by adjusting ε in Adam.
problem Improving performance of Adam optimizer.
method Proposes EAdam optimizer by adjusting the position of ε in Adam.
result EAdam outperforms Adam on various tasks and models.
Adam-type optimizers show one-sided convergence in GAN training, not reaching critical points.
problem Theoretical understanding of Adam-type optimizers in non-convex non-concave min-max optimization.
method Empirical and theoretical analysis of Adam-type algorithms' convergence in GAN training.
result Adam-type algorithms converge to one-sided first order stationary points under the one-sided MVI condition.
This work analyzes convergence and generalization of RMSProp and ADAM in non-convex optimization.
problem Understanding and improving convergence and generalization of adaptive gradient algorithms.
method Theoretical proofs and empirical experiments on autoencoders and VGG-9.
result ADAM can outperform NAG at high momentum parameters but NAG is better at low values.
New framework for Adam-type algorithms with constant β1, improving regret analysis.
problem Theoretical vs. practical use of Adam and variants with constant β1.
method Proposed a novel framework to derive optimal, data-dependent regret bounds with constant β1.
result Optimal, data-dependent regret bounds with constant β1 are achievable without further assumptions.
AdamNX improves Adam's stability by adjusting its learning rate.
problem Adam's tendency to converge to non-flat minima in large-scale models.
method Proposes a novel exponential decay mechanism for Adam's second-order moment estimate.
result AdamNX outperforms Adam and its variants in stability and performance.
Adam's convergence rate in nonsmooth nonconvex optimization is analyzed.
problem Understanding Adam's convergence behavior in nonsmooth nonconvex optimization.
method Finite-time analysis of Adam's classical form.
result Proven convergence rate of 1 / T 2 / 13 1/T^{2/13} 1/ T 2/13 for nonsmooth nonconvex optimization. Adam avoids simplicity bias in neural networks, leading to better generalization.
problem Simplicity bias in neural networks trained with SGD.
method Comparison of Adam and GD on binary classification tasks with Gaussian data.
result Adam leads to richer and more diverse features, improving generalization.
Deep neural network learning can be formulated as a non-convex optimization problem. Existing optimization algorithms, e.g., Adam, can learn the models fast, but may get stuck in local optima easily. In this paper, we introduce a novel optimization algorithm, namely GADAM (Genetic-Evolutionary Adam). GADAM learns deep …
Adam optimization algorithm can have non-zero average regret under certain conditions.
problem Non-zero average regret in Adam optimization algorithm.
method Used a three-periodic sequence of linear functions on [-1,1] with slopes c, -1, -1, and analyzed Adam variants.
result Adam optimization algorithm can have non-zero average regret under certain conditions.
New optimizer G-AdaGrad improves upon AdaGrad for non-convex machine learning problems.
problem Solving non-convex machine learning problems efficiently.
method Proposes a new optimizer G-AdaGrad and analyzes its convergence using state-space models.
result Empirical results show G-AdaGrad performs better than AdaGrad and Adam.
New Adam optimizer generalized for manifold training of neural networks.
problem Lack of clear physical intuition and difficulty in generalizing Adam optimizer to manifolds.
method Leverages the global tangent space representation of manifolds to perform Adam optimizer steps.
result Significant speed-ups in transformer training with orthogonality constraints.
AdaShift solves non-convergence issue of Adam by decorrelating gradient and second-moment terms.
problem Non-convergence of adaptive learning rate methods like Adam.
method AdaShift decorrelates gradient and second-moment terms by temporal shifting.
result AdaShift solves non-convergence problem of Adam and maintains competitive performance.
Adambs adapts Adam to prioritize important training examples.
problem Training deep learning models efficiently with varying example importance.
method Adapt Adam using a bandit sampling approach to prioritize examples based on their importance.
result Adambs improves convergence rate compared to Adam in some cases.
Adam converges with high probability under unconstrained non-convex smooth stochastic optimizations.
problem Theoretical limitations of Adam's convergence under unconstrained non-convex smooth stochastic optimizations.
method Deep analysis of Adam's convergence rate under affine variance noise, without bounded gradient assumptions.
result Adam converges to the stationary point with a high probability rate of $\mathcal{O}\left({
m poly}(\log T)/\sqrt{T}
ight)$ .
Adam can lead to worse test errors than GD in deep learning, especially with over-parameterized networks.
problem Adam's inferior generalization performance compared to GD in deep learning optimization.
method Theoretical analysis of Adam and gradient descent in over-parameterized two-layer convolutional neural networks.
result Adam and GD can converge to different global solutions with different generalization errors in nonconvex optimization landscapes.
MaxVA improves Adam's step sizes by maximizing gradient variance.
problem Unstable or extreme adaptive learning rates in Adam.
method Maximizing the variance of gradient coordinates in Adam's running mean of squared gradients.
result Faster adaptation and more desirable convergence behaviors than Adam.
We provide a simple convergence proof for Adam and Adagrad.
problem Smooth objective functions with bounded gradients.
method Simple proof covering Adam and Adagrad.
result Explicit upper-bound on the squared norm of the objective gradient.
Improves training GANs by escaping limit cycles.
problem Limit cycling behavior in training GANs.
method Predictive Centripetal Acceleration Algorithm (PCAA) combined with Adam.
result PCAA improves convergence rates and effectively trains GANs.
Adam's bias shifts from full-batch to max-margin of different norms for separable data.
problem Understanding Adam's implicit bias in the incremental batch setting.
method Analyzing incremental Adam on linearly separable data, constructing datasets, and using a proxy algorithm.
result Incremental Adam can converge to different max-margin classifiers depending on the dataset and batching scheme.
New framework adds trend information to Adam-type optimizers for faster convergence.
problem Efficiently optimizing complex cost surfaces in deep learning.
method Integrates trend information into Adam-type optimizers for adaptive step size and gradients.
result Framework outperforms conventional Adam and AMSGrad methods on various datasets.
HyperAdam learns to optimize neural networks by combining Adam's updates with varying decay rates.
problem Limitation of learned black-box optimizers in generalization ability.
method HyperAdam combines learned and traditional Adam optimizer, adaptively learning weights and decay rates.
result HyperAdam outperforms traditional optimizers in various network training tasks.
ADOPT optimizes Adam to converge with any β2 without bounded noise.
problem Non-convergence of Adam optimization algorithm.
method ADOPT removes current gradient from second moment estimate and changes momentum update order.
result ADOPT achieves optimal convergence rate of O(1 / √T) with any β2.
Bayesian Adam method improves neural network uncertainty estimation.
problem Improving uncertainty estimation in neural networks.
method Probabilistic interpretation of adaptive optimisation algorithms.
result Bayesian Adam method correctly relates learnt uncertainties to weights' predictive capabilities.
AdamQLR optimizes Adam with K-FAC heuristics, achieving comparable performance to tuned benchmarks.
problem Improving the performance of Adam optimizers with stabilizing heuristics.
method Combining Adam's update directions with K-FAC's heuristics (damping and learning rate selection).
result Untuned AdamQLR can achieve comparable performance to tuned benchmarks.
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
Proposes QHM and QHAdam for deep learning, improving training.
problem Improving training in deep learning.
method Introduces QHM and QHAdam, simple modifications of momentum SGD and Adam.
result Significantly improved training in various settings, including WMT16 EN-DE.
MDA optimizer performs similarly to SGD+M in CV and Adam in NLP.
problem Performance degradation due to choosing the wrong optimizer.
method Modernized Dual Averaging (MDA) optimizer, inspired by dual averaging.
result MDA performs as well as SGD+M in CV and as Adam in NLP.