A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Adam's generalization performance is improved by batch size and weight decay in neural networks.
Adam optimizer converges to zeros of a new vector field, not just gradient zeros.
Paper studies Adam's convergence under relaxed assumptions, proving a rate of O(poly(log T)/sqrt(T)).
In this paper, we investigate the popular deep learning optimization routine, Adam, from the perspective of statistical moments. While Adam is an adaptive lower-order moment based (of the stochastic gradient) method, we propose an extension namely, HAdam, which uses higher order moments of the stochastic gradient. Our …
Adam converges with high probability under unconstrained non-convex smooth stochastic optimizations.
Hill-ADAM optimizes loss landscapes by exploring state space deterministically.
The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an esti…
GeoAdaLer enhances geometric understanding of Adam for stochastic optimization.
Adam optimizes linear classifiers with separable data.
Study on Adam-family methods for nonsmooth optimization with convergence guarantees.
Adam and RMSProp are two of the most influential adaptive stochastic algorithms for training deep neural networks, which have been pointed out to be divergent even in the convex setting via a few simple counterexamples. Many attempts, such as decreasing an adaptive learning rate, adopting a big batch size, incorporatin…
Proposes VSGD optimizer combining probabilistic and gradient-based methods.
Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this wo…
Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, espec…
Improved stochastic gradient estimation for deep learning in high dimensions.
AdaX improves Adam by exponentially accumulating past gradients, leading to better performance in machine learning tasks.
Adam is a popular variant of stochastic gradient descent for finding a local minimizer of a function. In the constant stepsize regime, assuming that the objective function is differentiable and non-convex, we establish the convergence in the long run of the iterates to a stationary point under a stability condition. Th…
MDA optimizer performs similarly to SGD+M in CV and Adam in NLP.
Quantized Adam reduces communication cost in deep learning training.
This research explains why SGD generalizes better than ADAM in deep learning.
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
AdaSGD combines SGD and Adam benefits, eliminating the need for transition.
A new method Expectigrad improves on Adam and RMSProp by reducing divergence and improving performance.
This paper explains why Adam generalizes worse than SGD by analyzing its components.
SA-Solver improves stochastic sampling from DPMs.
Novel Adam-family method with decoupled weight decay for training neural networks.
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with mom…
This paper uses a geometric approach to understand how normalization layers affect neural network optimization.
Adam can lead to worse test errors than GD in deep learning, especially with over-parameterized networks.
MaxVA improves Adam's step sizes by maximizing gradient variance.
This paper improves Adam's performance in machine learning tasks.
Adam avoids simplicity bias in neural networks, leading to better generalization.
New optimizer Eve uses examplewise gradients for better second-moment estimates.
The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large …
The paper provides convergence guarantees for VAEs using SGD and Adam.
AdamNX improves Adam's stability by adjusting its learning rate.
RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or …
Deep neural networks are traditionally trained using human-designed stochastic optimization algorithms, such as SGD and Adam. Recently, the approach of learning to optimize network parameters has emerged as a promising research topic. However, these learned black-box optimizers sometimes do not fully utilize the experi…
Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameter…
Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the gene…
Adam-type optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning. Such methods are appealing due to the capability on large-scale sparse datasets with high computational efficiency. In this paper, we pres…
New adaptive stepsize method for stochastic approximation converges to target point.
Although ADAM is a very popular algorithm for optimizing the weights of neural networks, it has been recently shown that it can diverge even in simple convex optimization examples. Several variants of ADAM have been proposed to circumvent this convergence issue. In this work, we study the ADAM algorithm for smooth nonc…
This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their popularity in trainin…
Shampoo achieves higher token efficiency than Muon in language models.
Adaptive gradient methods such as Adam have been shown to be very effective for training deep neural networks (DNNs) by tracking the second moment of gradients to compute the individual learning rates. Differently from existing methods, we make use of the most recent first moment of gradients to compute the individual …
Hamiltonian Monte Carlo (HMC) is a popular Markov chain Monte Carlo (MCMC) algorithm that generates proposals for a Metropolis-Hastings algorithm by simulating the dynamics of a Hamiltonian system. However, HMC is sensitive to large time discretizations and performs poorly if there is a mismatch between the spatial geo…