On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

10 results for “warmup-stable-decay”

Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaystat.ML

Optimal learning rates decay to zero in easy tasks and maintain a warmup phase in hard tasks.

problem Optimizing learning rates under functional scaling laws for model training.

method Deriving optimal learning-rate schedules based on exponents

s

and

β

result Sharp phase transition between easy and hard tasks, with different decay behaviors.

learning rates functional scaling phase transition power decay warmup-stable-decay

2026-02-06abs ↗pdf ↗

GradPower: Powering Gradients for Faster Language Model Pre-Trainingcs.LG

GradPower speeds up language model training with minimal code changes.

problem Slower training of large language models.

method Elementwise sign-power transformation applied to gradients.

result Consistently lower terminal loss across various models and datasets.

language models gradient optimization faster training sign-power AdamPower

2025-05-30abs ↗pdf ↗

Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?cs.LG

SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.

problem Improving linear regression performance with signSGD under power-law random features.

method Analysis of signSGD risk under PLRF model, comparison with SGD, identification of unique effects.

result SignSGD can have a steeper compute-optimal slope than SGD in noisy regimes, especially with WSD schedule.

signSGD linear regression scaling laws power-law random features noise-reshaping

2026-03-02abs ↗pdf ↗

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecs.LG

WSD schedule improves model training efficiency by adapting learning rates dynamically.

problem Fixed compute budgets limit training efficiency of language models.

method Introduces a WSD schedule that uses a constant learning rate followed by a rapid decay phase.

result WSD schedule generates a non-traditional loss curve with stable and decay phases.

learning rates language models river valley landscape pretraining optimization

2024-10-07abs ↗pdf ↗

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscs.LG

ScheduleFree+ improves large language model training without schedules or learning rates.

problem Scaling up Schedule-Free Learning to large language models.

method Learning-rate-free and schedule-free method for training large language models.

result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.

Schedule-Free Learning Large Language Models Learning Rate Model Averaging Checkpoint Merging

2026-05-18abs ↗pdf ↗

WSqD: A Horizon-Free Learning Rate Schedule for Large Model Trainingcs.LG

WSqD extends learning rate schedules for large model training without fixed horizons.

problem Fixed learning rate schedules limit training horizon extension.

method WSqD replaces constant stable phase with a shifted inverse-square-root base, retaining linear cooldown.

result WSqD achieves minimax-optimal convergence rate and horizon-independence.

learning rate schedules horizon extension stochastic convex optimization last-iterate convergence language-model pretraining

2026-07-12abs ↗pdf ↗

Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcs.LG

New method SF-AdamW trains large models without decay phases or memory overhead.

problem Inadequate fixed compute budgets for large-scale training.

method Schedule-Free (SF) method revisited and refined.

result SF-AdamW effectively navigates loss landscape without decay phases or memory overhead.

language models training schedule-free flexibility empirical performance

2025-07-14abs ↗pdf ↗

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescs.LG

The paper presents a multi-power law for predicting loss curves across different learning rate schedules.

problem Understanding and optimizing the relationship between model performance and hyperparameters, especially learning rates.

method Proposes a multi-power law that combines power laws based on the sum of learning rates and additional laws for loss reduction due to decay.

result The multi-power law accurately predicts loss curves for unseen learning rate schedules and finds a schedule that outperforms cosine learning rate.

learning rate pretraining loss multi-power law loss curve prediction hyperparameter optimization

2025-03-17abs ↗pdf ↗

Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedulescs.LG

We discover scaling laws for kernel regression loss under various learning rate schedules.

problem Understanding loss dynamics and learning rate schedules in kernel regression.

method Theoretical analysis of stochastic gradient descent on a power-law kernel regression model.

result Established a Functional Scaling Law (FSL) capturing the full loss trajectory under arbitrary learning rate schedules.

Kernel Regression Learning Rate Schedules Functional Scaling Law Stochastic Gradient Descent Power-Law Model

2025-09-23abs ↗pdf ↗

Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Modelcond-mat.dis-nn

We find optimal learning rate schedules for a random feature model.

problem Choosing optimal learning rates for deep learning models.

method We analyze a powerlaw random feature model trained with SGD, considering optimal schedules as numerical and analytical problems.

result We discover two regimes: easy and hard phases, with different optimal learning rate schedules.

learning rate random feature model optimal schedule powerlaw stochastic gradient descent

2026-02-04abs ↗pdf ↗