On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

2 results for “AdEMAMix”

The AdEMAMix Optimizer: Better, Faster, Oldercs.LG

AdEMAMix optimizer improves model performance and convergence speed.

problem Suboptimal use of single EMA in momentum-based optimizers.

method Proposes AdEMAMix, a modified Adam optimizer with a mixture of two EMAs.

result Empirically shows gradients can remain relevant for tens of thousands of steps, leading to faster convergence and lower minima.

AdEMAMix optimizer momentum EMAs gradient relevance

2024-09-05abs ↗pdf ↗

Logarithmic-time Schedules for Scaling Language Models with Momentumstat.ML

Logarithmic-time schedules boost large-scale language model training efficiency.

problem Improving performance and efficiency in large-scale language model training.

method Designing time-varying hyperparameters

(β_1, β_2, λ)

for AdamW, specifically logarithmic-time scheduling with damping mechanisms.

result ADANA optimizer achieves up to 40% compute efficiency compared to tuned AdamW, with gains persisting as model scale increases.

language models AdamW logarithmic-time scheduling performance gains compute efficiency

2026-02-05abs ↗pdf ↗