On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

169,051 papers · 148 categories

Trend · papers per month

count % of corpus

2 results for “AdaShift”

AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methodscs.LG

AdaShift solves non-convergence issue of Adam by decorrelating gradient and second-moment terms.

problem Non-convergence of adaptive learning rate methods like Adam.

method AdaShift decorrelates gradient and second-moment terms by temporal shifting.

result AdaShift solves non-convergence problem of Adam and maintains competitive performance.

AdaShift Adam non-convergence adaptive learning rate gradient correlation

2018-09-29abs ↗pdf ↗

An Adaptive and Momental Bound Method for Stochastic Learningcs.LG

New method AdaMod stabilizes deep neural network training by limiting adaptive learning rates.

problem Adaptive learning rates can produce extremely large values at the start of training, hindering learning.

method AdaMod uses adaptive and momental upper bounds to restrict learning rates dynamically.

result AdaMod eliminates large learning rates and improves training on complex networks.

deep learning adaptive learning rates momentum stochastic optimization neural networks

2019-10-27abs ↗pdf ↗