Adam optimizes non-convex functions, converging to critical points.
problem Finding local minima of non-convex functions.
method Continuous-time model of Adam, ODE approximation, decreasing stepsize.
result Adam converges to critical points of non-convex functions.
Differential K-theory gets a λ-ring structure.
problem Establishing a λ-ring structure in differential K-theory. method Splitting principle for differential K-theory, Adams operations construction.
result Differential K0-ring admits a λ-ring structure. Proves a conjecture for a specific group using spectral sequences and homology.
problem Proves the Gromov-Lawson-Rosenberg Conjecture for the group Z/4xZ/4.
method Used the Adams spectral sequence and detection theorems to compute connective real k-homology.
result Determines differentials of the Adams spectral sequence and studies the cap structure of relevant sub-hopf algebras.
This research explains why SGD generalizes better than ADAM in deep learning.
problem Understanding the generalization gap between SGD and ADAM in deep learning.
method Analyzing local convergence behaviors through Levy-driven stochastic differential equations (SDEs).
result SGD is more locally unstable and better escapes from sharp minima to flatter ones, leading to better generalization.
Paper explores challenges in training PINNs and loss landscape effects.
problem Challenges in training Physics-Informed Neural Networks (PINNs) due to loss landscape issues.
method Examined gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, and introduced NysNewton-CG (NNCG).
result Adam+L-BFGS outperforms other optimizers, and NysNewton-CG significantly improves PINN performance.
The 61-stem in stable homotopy groups of spheres is trivial.
problem Proving the triviality of the 61-stem in stable homotopy groups of spheres.
method Computation of homotopy groups of spheres, introducing a new technique based on Kahn-Priddy theorems.
result The 2-primary π61 is zero. Sharp inequalities on Siegel domains and complex hyperbolic spaces established.
problem Establishing inequalities on complex hyperbolic spaces and Siegel domains.
method Helgason-Fourier analysis, Kunze-Stein phenomenon, factorization theorem.
result Sharp Hardy-Adams and Adams type inequalities on Sobolev spaces of any positive fractional order on complex hyperbolic spaces.
Differentially private GANs improve image privacy without significant quality loss.
problem Anonymizing image data sets while maintaining image quality.
method Training GANs with differential privacy on MNIST, analyzing privacy-utility trade-offs and explaining optimization methods.
result An increasing privacy budget adds little to generated image quality, revealing a saturated training regime.
Adam's hyperparameters implicitly regularize solutions, penalizing or impeding loss gradients' norms.
problem Implicit regularization in Adam's hyperparameters and training stage.
method Backward error analysis and ODE approximations to study Adam's behavior.
result Adam's implicit regularization depends on hyperparameters and training stage, involving different norms.
New optimizer G-AdaGrad improves upon AdaGrad for non-convex machine learning problems.
problem Solving non-convex machine learning problems efficiently.
method Proposes a new optimizer G-AdaGrad and analyzes its convergence using state-space models.
result Empirical results show G-AdaGrad performs better than AdaGrad and Adam.
SA-Solver improves stochastic sampling from DPMs.
problem Efficient sampling from Diffusion Probabilistic Models (DPMs) is time-consuming.
method Proposes SA-Solver, an improved stochastic Adams method for solving diffusion SDE.
result SA-Solver achieves improved or comparable performance compared to SOTA methods for few-step sampling.
Neural circuits integrate continuous dynamics efficiently.
problem Efficiently integrating continuous neural dynamics for simulation and learning.
method Compact neural circuits for Runge-Kutta and Adams-Bashforth-Moulton methods.
result Equivalence of neural and numerical integration for polynomial systems.
ND-Adam improves generalization of Adam by controlling weight updates.
problem Adam leads to worse generalization than SGD, especially for deep neural networks.
method Developed ND-Adam, a variant of Adam that controls weight updates more precisely.
result ND-Adam significantly improves generalization performance compared to Adam.
Improved deep learning optimizers using adaptive stepsize.
problem Improving the performance of deep learning optimizers.
method Adapts stepsize directly with the loss function to make progress on loss.
result Enhanced optimizers outperform Adam and Momentum optimizers without increased computational cost.
Paper improves deep learning privacy with new f-differential privacy framework.
problem Training neural networks on sensitive data while maintaining privacy.
method Introduced and analyzed f-differential privacy for neural networks training. result Improved privacy guarantees for neural networks training without sacrificing accuracy.
Hill-ADAM optimizes loss landscapes by exploring state space deterministically.
problem Escaping local minima in loss landscapes.
method Hill-ADAM alternates between minimizing and maximizing error to explore the loss space.
result Hill-ADAM finds the global minimum state in loss landscapes.
EAdam optimizer improves performance by adjusting ε in Adam.
problem Improving performance of Adam optimizer.
method Proposes EAdam optimizer by adjusting the position of ε in Adam.
result EAdam outperforms Adam on various tasks and models.
Adam's generalization performance is improved by batch size and weight decay in neural networks.
problem Understanding how batch size and weight decay affect Adam's generalization in neural networks.
method Theoretical analysis of two-layer over-parameterized CNNs on image data.
result Adam's mini-batch variants can achieve near-zero test error, unlike full-batch Adam.
AdamS uses momentum as a denominator to optimize LLMs efficiently.
problem Optimizing large language models (LLMs) with efficient and effective methods.
method AdamS introduces a novel denominator based on the root of the weighted sum of squares of momentum and current gradient.
result AdamS achieves superior optimization performance with minimal memory and compute requirements.
ADAM optimizer analyzed for sign, magnitude, and variance of stochastic gradients.
problem Understanding and improving ADAM optimizer performance and generalization.
method Interpreted ADAM as sign and variance aspects, analyzed separately, and introduced a new method.
result Sign aspect of ADAM is problematic for generalization; variance adaptation improves performance.
Adam optimizes linear classifiers with separable data.
problem Understanding Adam's implicit bias in linear logistic regression.
method Study of Adam's behavior on linearly separable data.
result Adam converges to a linear classifier with maximum ℓ∞-margin. Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.
problem Inefficiencies in ad-hoc learning rate tuning for optimization algorithms.
method Developed AdaGrad++ and Adam++ without predefined learning rates and proved their convergence.
result AdaGrad++ and Adam++ achieve comparable convergence rates to AdaGrad and Adam respectively.
Develops differential KO-theory with constructions and applications.
problem Constructing differential refinements in KO-theory.
method Explicit constructions and refinements of spectral sequences.
result Identification of differentials in differential KO-theory.
A new memory-efficient Adam variant reduces second moments when feasible.
problem Memory constraints in training machine learning models.
method Signal-to-Noise Ratio (SNR) analysis to identify dimensions where second moments can be replaced by means.
result Memory-efficient Adam variant (SlimAdam) matches performance and stability of Adam while saving up to 98% of second moments.
Adam optimizer converges to zeros of a new vector field, not just gradient zeros.
problem Prove convergence rates for Adam optimizer in simple quadratic optimization problems.
method Introduced Adam vector field to analyze Adam optimizer's convergence.
result Established optimal convergence rates for Adam optimizer.
AdaX improves Adam by exponentially accumulating past gradients, leading to better performance in machine learning tasks.
problem Adam's fast convergence can lead to local minimums in non-convex problems.
method AdaX exponentially accumulates past gradients to adaptively tune the learning rate.
result AdaX outperforms Adam in various machine learning tasks, including computer vision and natural language processing.
AdaSGD combines SGD and Adam benefits, eliminating the need for transition.
problem Understanding when to transition from Adam to SGD for optimal performance.
method Adapting a single global learning rate for SGD (AdaSGD).
result AdaSGD combines the benefits of both SGD and Adam, improving convergence and generalization.
Improved Adam for time series forecasting with distributional drift.
problem Non-stationary data challenges Adam's effectiveness.
method Proposed TS_Adam, removing Adam's second-order bias correction.
result TS_Adam achieves 12.8% reduction in MSE and 5.7% in MAE on ETT datasets.
Adam optimizer fails to stay close to optimal point under certain conditions.
problem Adam optimizer's tendency to deviate from the optimal point in training neural networks.
method Analyzed Adam's behavior in convex regions and proposed a new algorithm to correct this.
result Adam optimizer cannot stay close to the optimal point when effective learning rate exceeds a certain bound.
Extends Adams' theorem to periodic cohomology.
problem Proving Adams' theorem for periodic cohomology.
method Adapting Adams' approach to periodic cohomology.
result Conjecture proven in a special case.
DualAdam improves generalization of Adam by integrating its update mechanisms.
problem Adam's tendency to converge to sharp minima leading to suboptimal generalization.
method DualAdam combines Adam and inverse Adam's update mechanisms to enhance generalization.
result DualAdam outperforms Adam and state-of-the-art variants in generalization performance.
Enhanced Adam uses higher-order moments for better performance.
problem Improving the performance of Adam optimization algorithm.
method Proposes HAdam, an extension of Adam using higher-order moments of the stochastic gradient.
result Higher-order moments of the stochastic gradient can lead to better performance than vanilla Adam.
Adam converges to stationary points under relaxed conditions.
problem Understanding and proving convergence of Adam under realistic assumptions.
method New proof of boundedness of gradients and variance-reduced Adam.
result Adam converges to ε-stationary points with O(ε⁻⁴) gradient complexity under realistic conditions.
Nostalgic Adam improves long-term memory in adaptive learning rate algorithms.
problem Lack of long-term memory in Adam-like algorithms can lead to performance issues and divergence.
method Proposes Nostalgic Adam (NosAdam), which weights more past gradients, theoretically ensuring convergence.
result NosAdam can be a fix for Adam's non-convergence issues and is a promising alternative.
New methods improve neural network training by sampling from target distributions.
problem Optimization procedures like SGD drive parameters to local minima.
method Thermodynamic parameterization using discretized stochastic differential equations.
result Partitioned numerical algorithms converge faster and more robustly.
Laprop separates Adam's momentum and adaptivity to improve stability and speed.
problem Unnecessary coupling between Adam's momentum and adaptivity leads to instability and divergence.
method Proposes Laprop, a method that decouples momentum and adaptivity.
result Laprop consistently improves speed and stability over Adam on various tasks.
Untuned warmup of Adam performs similarly to RAdam in practical settings.
problem The stability of adaptive optimization algorithms like Adam requires a warmup schedule.
method Provided alternative explanation for warmup necessity based on update term magnitude.
result Untuned warmup of Adam is as effective as RAdam in typical practical settings.
Novel Adam-family method with decoupled weight decay for training neural networks.
problem Training nonsmooth neural networks with weight decay.
method Proposes a novel Adam-family method with decoupled weight decay, establishing convergence properties and demonstrating superior performance.
result Asymptotically approximates SGD and enhances generalization performance.
AdamCB optimizes neural network training by adaptively selecting samples.
problem Inefficient convergence due to unequal influence of different data samples.
method Integrates combinatorial bandit techniques into Adam to adaptively select samples.
result AdamCB achieves faster convergence and better performance than existing methods.
APMSqueeze improves Adam for faster training with less communication.
problem Adam's inefficiency in parallel training due to gradient compression limitations.
method Error-compensated gradient compression for Adam-preconditioned Momentum SGD.
result Significantly reduces communication time per epoch without sacrificing accuracy.
AdaShift solves non-convergence issue of Adam by decorrelating gradient and second-moment terms.
problem Non-convergence of adaptive learning rate methods like Adam.
method AdaShift decorrelates gradient and second-moment terms by temporal shifting.
result AdaShift solves non-convergence problem of Adam and maintains competitive performance.
Paper studies Adam's convergence under relaxed assumptions, proving a rate of O(poly(log T)/sqrt(T)).
problem Understanding Adam's convergence in non-convex, stochastic optimization with unbounded gradients and noise.
method Introduced a comprehensive noise model and used it to prove Adam's convergence rate.
result Adam finds a stationary point with a rate of O(poly(log T)/sqrt(T)) in high probability.
A new sufficient condition ensures convergence of Adam and RMSProp in non-convex settings.
problem Adam and RMSProp diverge in convex settings, despite being influential for deep learning.
method Introduces a sufficient condition based on base learning rate and historical second-order moments.
result Guarantees global convergence of generic Adam/RMSProp in non-convex optimization.
New algorithms improve generalization of Adam and AdamW optimizers.
problem Adam and AdamW optimizers generalize worse than SGD.
method Algorithmic stability analysis and clever momentum-based SGD integration.
result HomeAdam(W) algorithms achieve better generalization and faster convergence.
Adam's convergence rate in nonsmooth nonconvex optimization is analyzed.
problem Understanding Adam's convergence behavior in nonsmooth nonconvex optimization.
method Finite-time analysis of Adam's classical form.
result Proven convergence rate of 1/T2/13 for nonsmooth nonconvex optimization. Adam optimization algorithm can have non-zero average regret under certain conditions.
problem Non-zero average regret in Adam optimization algorithm.
method Used a three-periodic sequence of linear functions on [-1,1] with slopes c, -1, -1, and analyzed Adam variants.
result Adam optimization algorithm can have non-zero average regret under certain conditions.
This work analyzes Adam's preconditioning effect on quadratic functions and quantifies its impact on condition number.
problem Understanding and quantifying the preconditioning effect of Adam to alleviate ill-conditioning in gradient descent.
method Detailed analysis of Adam's preconditioning effect for quadratic functions, including empirical evidence.
result Adam can mitigate the condition number but at a dimension-dependent cost, with specific bounds for different types of Hessians.
GRU models with Adam optimizer outperform other combinations in stock market forecasting.
problem Comparing optimization techniques for time series forecasting in LSTM and GRU networks.
method Examined Adam and Nesterov Accelerated Gradient (NAG) on LSTM and GRU models for stock market forecasting.
result GRU models with Adam optimizer produced the lowest RMSE and outperformed other combinations.