On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,982 papers · 148 categories

Trend · papers per month

count % of corpus

6 results for “Nadam”

On the Convergence of Adam and Beyondcs.LG

Adam and similar methods fail to converge in some deep learning tasks.

problem Adam and similar methods fail to converge to optimal solutions in some deep learning tasks.

method Analysis of gradient updates scaled by square roots of exponential moving averages of squared past gradients. Proposed new variants of Adam with `long-term memory'.

result New variants of Adam algorithms fix convergence issues and often lead to improved empirical performance.

Adam RMSProp Adadelta Nadam Convergence

2019-04-19abs ↗pdf ↗

On the Convergence of Adam, Revisitedcs.LG

Adam optimization algorithm can have non-zero average regret under certain conditions.

problem Non-zero average regret in Adam optimization algorithm.

method Used a three-periodic sequence of linear functions on [-1,1] with slopes c, -1, -1, and analyzed Adam variants.

result Adam optimization algorithm can have non-zero average regret under certain conditions.

Adam optimization regret analysis moment decay parameters three-periodic sequence linear functions

2026-07-03abs ↗pdf ↗

An Adaptive Remote Stochastic Gradient Method for Training Neural Networkscs.LG

ARSG method improves neural network training speed and generalization.

problem Training neural networks efficiently and accurately.

method Adaptive remote stochastic gradient method combining adaptive techniques.

result ARSG achieves faster convergence and better generalization compared to popular methods.

neural networks stochastic gradient adaptive methods convergence rate generalization

2019-05-04abs ↗pdf ↗

A Rod Flow Model for Adam at the Edge of Stabilitycs.LG

Rod flow models Adam's behavior at the edge of stability.

problem Modeling adaptive gradient methods like Adam at the edge of stability.

method Extended rod flow to Adam, considering parameters, first moment, and second moment as variables.

result Rod flow accurately tracks Adam's behavior through the edge-of-stability regime.

Rod flow Adam Edge of stability Momentum methods Machine learning

2026-05-07abs ↗pdf ↗

Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activationcs.LG

SGD methods fail to converge to global minimizers in deep neural networks with ReLU activation.

problem Failure of SGD methods to converge to global minimizers in deep neural networks.

method Stochastic Gradient Descent (SGD) and its variants like Adam, RMSProp, etc.

result SGD methods fail to converge to global minimizers with high probability in deep neural networks with ReLU activation.

deep learning SGD ReLUs non-convergence global minimizers

2024-10-14abs ↗pdf ↗

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizerscs.LG

A higher-order Runge-Kutta optimizer performs poorly compared to Adam when evaluated fairly.

problem Evaluating the performance of adaptive Runge-Kutta optimizers under strict conditions.

method Built and evaluated a representative Adam variant using a Bogacki-Shampine 3(2) RK pair, FSAL reuse, and local-error step control.

result The adaptive nature of the RK optimizer is illusory; it behaves like a fixed-step optimizer with gradient averaging.

Runge-Kutta optimization Adam training loss generalization

2026-07-16abs ↗pdf ↗