Hill-ADAM optimizes loss landscapes by exploring state space deterministically.
problem Escaping local minima in loss landscapes.
method Hill-ADAM alternates between minimizing and maximizing error to explore the loss space.
result Hill-ADAM finds the global minimum state in loss landscapes.
EAdam optimizer improves performance by adjusting ε in Adam.
problem Improving performance of Adam optimizer.
method Proposes EAdam optimizer by adjusting the position of ε in Adam.
result EAdam outperforms Adam on various tasks and models.
Adam's generalization performance is improved by batch size and weight decay in neural networks.
problem Understanding how batch size and weight decay affect Adam's generalization in neural networks.
method Theoretical analysis of two-layer over-parameterized CNNs on image data.
result Adam's mini-batch variants can achieve near-zero test error, unlike full-batch Adam.
AdamS uses momentum as a denominator to optimize LLMs efficiently.
problem Optimizing large language models (LLMs) with efficient and effective methods.
method AdamS introduces a novel denominator based on the root of the weighted sum of squares of momentum and current gradient.
result AdamS achieves superior optimization performance with minimal memory and compute requirements.
Adam optimizes linear classifiers with separable data.
problem Understanding Adam's implicit bias in linear logistic regression.
method Study of Adam's behavior on linearly separable data.
result Adam converges to a linear classifier with maximum ℓ∞-margin. Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.
problem Inefficiencies in ad-hoc learning rate tuning for optimization algorithms.
method Developed AdaGrad++ and Adam++ without predefined learning rates and proved their convergence.
result AdaGrad++ and Adam++ achieve comparable convergence rates to AdaGrad and Adam respectively.
A new memory-efficient Adam variant reduces second moments when feasible.
problem Memory constraints in training machine learning models.
method Signal-to-Noise Ratio (SNR) analysis to identify dimensions where second moments can be replaced by means.
result Memory-efficient Adam variant (SlimAdam) matches performance and stability of Adam while saving up to 98% of second moments.
Adam optimizer converges to zeros of a new vector field, not just gradient zeros.
problem Prove convergence rates for Adam optimizer in simple quadratic optimization problems.
method Introduced Adam vector field to analyze Adam optimizer's convergence.
result Established optimal convergence rates for Adam optimizer.
AdaX improves Adam by exponentially accumulating past gradients, leading to better performance in machine learning tasks.
problem Adam's fast convergence can lead to local minimums in non-convex problems.
method AdaX exponentially accumulates past gradients to adaptively tune the learning rate.
result AdaX outperforms Adam in various machine learning tasks, including computer vision and natural language processing.
AdaSGD combines SGD and Adam benefits, eliminating the need for transition.
problem Understanding when to transition from Adam to SGD for optimal performance.
method Adapting a single global learning rate for SGD (AdaSGD).
result AdaSGD combines the benefits of both SGD and Adam, improving convergence and generalization.
Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this wo…
Improved Adam for time series forecasting with distributional drift.
problem Non-stationary data challenges Adam's effectiveness.
method Proposed TS_Adam, removing Adam's second-order bias correction.
result TS_Adam achieves 12.8% reduction in MSE and 5.7% in MAE on ETT datasets.
Extends Adams' theorem to periodic cohomology.
problem Proving Adams' theorem for periodic cohomology.
method Adapting Adams' approach to periodic cohomology.
result Conjecture proven in a special case.
DualAdam improves generalization of Adam by integrating its update mechanisms.
problem Adam's tendency to converge to sharp minima leading to suboptimal generalization.
method DualAdam combines Adam and inverse Adam's update mechanisms to enhance generalization.
result DualAdam outperforms Adam and state-of-the-art variants in generalization performance.
Adam converges to stationary points under relaxed conditions.
problem Understanding and proving convergence of Adam under realistic assumptions.
method New proof of boundedness of gradients and variance-reduced Adam.
result Adam converges to ε-stationary points with O(ε⁻⁴) gradient complexity under realistic conditions.
The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an esti…
Novel Adam-family method with decoupled weight decay for training neural networks.
problem Training nonsmooth neural networks with weight decay.
method Proposes a novel Adam-family method with decoupled weight decay, establishing convergence properties and demonstrating superior performance.
result Asymptotically approximates SGD and enhances generalization performance.
AdamCB optimizes neural network training by adaptively selecting samples.
problem Inefficient convergence due to unequal influence of different data samples.
method Integrates combinatorial bandit techniques into Adam to adaptively select samples.
result AdamCB achieves faster convergence and better performance than existing methods.
APMSqueeze improves Adam for faster training with less communication.
problem Adam's inefficiency in parallel training due to gradient compression limitations.
method Error-compensated gradient compression for Adam-preconditioned Momentum SGD.
result Significantly reduces communication time per epoch without sacrificing accuracy.
Paper studies Adam's convergence under relaxed assumptions, proving a rate of O(poly(log T)/sqrt(T)).
problem Understanding Adam's convergence in non-convex, stochastic optimization with unbounded gradients and noise.
method Introduced a comprehensive noise model and used it to prove Adam's convergence rate.
result Adam finds a stationary point with a rate of O(poly(log T)/sqrt(T)) in high probability.
New algorithms improve generalization of Adam and AdamW optimizers.
problem Adam and AdamW optimizers generalize worse than SGD.
method Algorithmic stability analysis and clever momentum-based SGD integration.
result HomeAdam(W) algorithms achieve better generalization and faster convergence.
Adam's convergence rate in nonsmooth nonconvex optimization is analyzed.
problem Understanding Adam's convergence behavior in nonsmooth nonconvex optimization.
method Finite-time analysis of Adam's classical form.
result Proven convergence rate of 1/T2/13 for nonsmooth nonconvex optimization. Adam optimization algorithm can have non-zero average regret under certain conditions.
problem Non-zero average regret in Adam optimization algorithm.
method Used a three-periodic sequence of linear functions on [-1,1] with slopes c, -1, -1, and analyzed Adam variants.
result Adam optimization algorithm can have non-zero average regret under certain conditions.
This work analyzes Adam's preconditioning effect on quadratic functions and quantifies its impact on condition number.
problem Understanding and quantifying the preconditioning effect of Adam to alleviate ill-conditioning in gradient descent.
method Detailed analysis of Adam's preconditioning effect for quadratic functions, including empirical evidence.
result Adam can mitigate the condition number but at a dimension-dependent cost, with specific bounds for different types of Hessians.
GRU models with Adam optimizer outperform other combinations in stock market forecasting.
problem Comparing optimization techniques for time series forecasting in LSTM and GRU networks.
method Examined Adam and Nesterov Accelerated Gradient (NAG) on LSTM and GRU models for stock market forecasting.
result GRU models with Adam optimizer produced the lowest RMSE and outperformed other combinations.
Adam avoids simplicity bias in neural networks, leading to better generalization.
problem Simplicity bias in neural networks trained with SGD.
method Comparison of Adam and GD on binary classification tasks with Gaussian data.
result Adam leads to richer and more diverse features, improving generalization.
Adam converges with high probability under unconstrained non-convex smooth stochastic optimizations.
problem Theoretical limitations of Adam's convergence under unconstrained non-convex smooth stochastic optimizations.
method Deep analysis of Adam's convergence rate under affine variance noise, without bounded gradient assumptions.
result Adam converges to the stationary point with a high probability rate of $\mathcal{O}\left({
m poly}(\log T)/\sqrt{T}
ight)$.
Adam can lead to worse test errors than GD in deep learning, especially with over-parameterized networks.
problem Adam's inferior generalization performance compared to GD in deep learning optimization.
method Theoretical analysis of Adam and gradient descent in over-parameterized two-layer convolutional neural networks.
result Adam and GD can converge to different global solutions with different generalization errors in nonconvex optimization landscapes.
The adaptive optimizer for training neural networks has continually evolved to overcome the limitations of the previously proposed adaptive methods. Recent studies have found the rare counterexamples that Adam cannot converge to the optimal point. Those counterexamples reveal the distortion of Adam due to a small secon…
MaxVA improves Adam's step sizes by maximizing gradient variance.
problem Unstable or extreme adaptive learning rates in Adam.
method Maximizing the variance of gradient coordinates in Adam's running mean of squared gradients.
result Faster adaptation and more desirable convergence behaviors than Adam.
Adam's bias shifts from full-batch to max-margin of different norms for separable data.
problem Understanding Adam's implicit bias in the incremental batch setting.
method Analyzing incremental Adam on linearly separable data, constructing datasets, and using a proxy algorithm.
result Incremental Adam can converge to different max-margin classifiers depending on the dataset and batching scheme.
First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of ``long-term memory" in Adam-like algorithms, which could hamper their performance and lead to divergence. In our s…
This research explains why SGD generalizes better than ADAM in deep learning.
problem Understanding the generalization gap between SGD and ADAM in deep learning.
method Analyzing local convergence behaviors through Levy-driven stochastic differential equations (SDEs).
result SGD is more locally unstable and better escapes from sharp minima to flatter ones, leading to better generalization.
Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of non-convergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide new insight into the non-convergence issue of Adam …
Adam and RMSProp are two of the most influential adaptive stochastic algorithms for training deep neural networks, which have been pointed out to be divergent even in the convex setting via a few simple counterexamples. Many attempts, such as decreasing an adaptive learning rate, adopting a big batch size, incorporatin…
The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.
problem Analyzing Adam and SGD in nonstationary optimization problems.
method Theoretical analysis of Adam and SGD under non-stationary stochastic objectives, separating two regimes.
result Characterizes the tradeoff between noise and drift in Adam and SGD, revealing when adaptive step-sizing is beneficial or harmful.
Adambs adapts Adam to prioritize important training examples.
problem Training deep learning models efficiently with varying example importance.
method Adapt Adam using a bandit sampling approach to prioritize examples based on their importance.
result Adambs improves convergence rate compared to Adam in some cases.
We provide a simple convergence proof for Adam and Adagrad.
problem Smooth objective functions with bounded gradients.
method Simple proof covering Adam and Adagrad.
result Explicit upper-bound on the squared norm of the objective gradient.
Adaptive optimization algorithms such as Adam are widely used in deep learning. The stability of such algorithms is often improved with a warmup schedule for the learning rate. Motivated by the difficulty of choosing and tuning warmup schedules, recent work proposes automatic variance rectification of Adam's adaptive l…
This paper improves Adam's performance in machine learning tasks.
problem Improving the generalization ability of adaptive gradient methods.
method Develops a control theoretic framework to propose AdamSSM, a new variant of Adam.
result AdamSSM improves generalization accuracy and convergence compared to recent adaptive gradient methods.
We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and…
This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.
problem Analyzing convergence of RMSProp and Adam in non-convex optimization with relaxed assumptions.
method Developed new convergence analyses for RMSProp and Adam, considering adaptive learning rates and affine noise variance.
result RMSProp and Adam converge to ε-stationary points with iteration complexities of O(ε^(-4)) under proper hyperparameters.
The study examines the dynamic behavior of RMSprop and Adam algorithms.
problem Understanding the training loss curve patterns of adaptive gradient algorithms.
method Careful numerical experiments and theoretical explanations using the signGD flow.
result Adam converges smoother and faster when momentum factors are close to each other.
In this paper, we investigate the popular deep learning optimization routine, Adam, from the perspective of statistical moments. While Adam is an adaptive lower-order moment based (of the stochastic gradient) method, we propose an extension namely, HAdam, which uses higher order moments of the stochastic gradient. Our …
ADOPT optimizes Adam to converge with any β2 without bounded noise.
problem Non-convergence of Adam optimization algorithm.
method ADOPT removes current gradient from second moment estimate and changes momentum update order.
result ADOPT achieves optimal convergence rate of O(1 / √T) with any β2.
Adam-type optimizers show one-sided convergence in GAN training, not reaching critical points.
problem Theoretical understanding of Adam-type optimizers in non-convex non-concave min-max optimization.
method Empirical and theoretical analysis of Adam-type algorithms' convergence in GAN training.
result Adam-type algorithms converge to one-sided first order stationary points under the one-sided MVI condition.
CAdam optimizes online learning by adapting to distribution shifts and noise.
problem Challenges in online learning data, including distribution shifts and noise, affect Adam's performance.
method CAdam uses a confidence-based approach to assess the consistency between momentum and gradients before updating parameters.
result CAdam outperforms other optimizers in various settings with distribution shift or noise.
Adam performs better with equal momentum parameters, revealing a gradient scale invariance principle.
problem Why Adam performs better with β1=β2. method Formalized gradient scale invariance and proved it for Adam with equal β1 and β2. result Adam becomes gradient scale invariant of first order if and only if β1=β2.