On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “large learning rate”

Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibilitycs.LG

Large learning rates enhance model robustness and compressibility.

problem Achieving robustness and resource-efficiency in machine learning models.

method Identifying and utilizing large learning rates as a facilitator for robustness and compressibility.

result Large learning rates produce desirable representation properties and compare favorably to other methods.

large learning rates robustness compressibility spurious correlations representation properties

2025-07-23abs ↗pdf ↗

The large learning rate phase of deep learning: the catapult mechanismstat.ML

New model explains deep learning performance at large learning rates.

problem Understanding deep learning performance at different learning rates.

method Developed neural networks with solvable training dynamics.

result Large learning rates lead to convergence to flatter minima.

deep learning learning rate neural networks training dynamics phase transition

2020-03-04abs ↗pdf ↗

Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscs.LG

Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is anne…

2019-07-10abs ↗pdf ↗

Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescs.LG

Large learning rates cause oscillations in NN weights that improve generalization.

problem Improving generalization of neural networks trained with large learning rates.

method Theoretical analysis and feature-noise data generation model.

result Oscillating SGD with large learning rates benefits NN generalization by effectively learning weak features.

stochastic gradient descent neural networks large learning rates generalization feature learning

2023-10-26abs ↗pdf ↗

The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcs.LG

Learning-rate schedules for large models match optimization theory closely, leading to better training.

problem Improving training of large models with optimal learning rates.

method Used a bound from non-smooth convex optimization theory to match learning-rate schedules with practical benefits.

result Extending the learning-rate schedule with optimal learning-rate and transferring it across schedules improves model training.

large models learning-rate scheduling convex optimization model training optimization theory

2025-01-31abs ↗pdf ↗

Implicit bias of deep linear networks in the large learning rate phasecs.LG

Study shows deep linear networks can converge to flatter minima at large learning rates.

problem Understanding the implicit bias of deep linear networks at large learning rates.

method Characterization of deep linear networks for binary classification using logistic loss in the large learning rate regime.

result Gradient descent iterates converge to a flatter minimum in the catapult phase for certain data separation conditions.

deep learning implicit bias large learning rate flatter minimum logistic loss

2020-11-25abs ↗pdf ↗

Large Learning Rates Improve Generalization: But How Large Are We Talking About?cs.LG

Large learning rates improve generalization, but optimal ranges are narrower than previously thought.

problem Optimizing learning rates for neural network training.

method Detailed exploration of learning rate ranges in a simplified setup, validating findings in a practical setting.

result Optimal learning rate ranges are significantly narrower than previously assumed.

neural networks learning rates generalization optimization

2023-11-19abs ↗pdf ↗

On the Surprising Effectiveness of Large Learning Rates under Standard Width Scalingcs.LG

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

problem Theoretical limits of large learning rates do not match practical network behavior.

method Fine-grained analysis of learning rates and network behavior under cross-entropy loss.

result There are two distinct sub-regimes of unstable learning rates, with a controlled divergence regime where features continue to evolve.

large learning rates standard parameterization cross-entropy loss finite-width phenomena infinite-width limits

2025-05-28abs ↗pdf ↗

Understanding the Generalization Benefits of Late Learning Rate Decaycs.LG

Large learning rates improve neural network generalization, study shows.

problem Understanding why large learning rates lead to better neural network generalization.

method Visual analysis of training and testing loss landscapes, introduction of a nonlinear model.

result Extended phase with large learning rates leads to near-optimal generalization.

neural networks generalization learning rate minima manifold training loss

2024-01-21abs ↗pdf ↗

On the Benefits of Large Learning Rates for Kernel Methodsstat.ML

Large learning rates improve kernel method performance.

problem Improving generalization in kernel methods with large learning rates.

method Analyzing the spectral decomposition of the solution to a quadratic objective in a separable Hilbert space.

result Large learning rates influence the spectral decomposition of the solution, leading to better generalization.

kernel methods large learning rates gradient descent spectral decomposition generalization

2022-02-28abs ↗pdf ↗

AdaScale SGD: A User-Friendly Algorithm for Distributed Trainingcs.LG

AdaScale SGD adapts learning rates for large-batch training efficiently.

problem Adapting learning rates for large-batch training to balance speed-ups and model quality.

method Adaptive learning rate adaptation based on gradient variance.

result AdaScale achieves reliable speed-ups for a wide range of batch sizes without degrading model quality.

AdaScale SGD Large-batch training Learning rate adaptation Gradient variance

2020-07-09abs ↗pdf ↗

Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcs.LG

Large learning rates lead to various implicit biases in nonconvex optimization.

problem Understanding the conditions under which large learning rates yield edge of stability, balancing, and catapult phenomena.

method Developed a global convergence theory for nonconvex functions without globally Lipschitz continuous gradient, focusing on functions with good regularity.

result These implicit biases are more likely to occur in functions with good regularity, and large learning rates favor flatter regions.

large learning rates nonconvex optimization edge of stability balancing catapult

2023-10-26abs ↗pdf ↗

SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Ratesstat.ML

SGD works well with large learning rates at the edge of stability.

problem Stochasticity at the edge of stability in deep learning.

method Sharp convergence guarantees for SGD with multiclass cross-entropy loss.

result SGD self-stabilizes, ensuring convergence with large learning rates.

SGD stochasticity edge of stability learning rates convergence

2026-06-29abs ↗pdf ↗

Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorizationstat.ML

Large learning rates prevent memorization in denoising score matching.

problem Memorization of training data in diffusion-based generative models.

method Investigating the role of large learning rates in the small-noise regime, proving that they prevent convergence to the empirical optimal score.

result Large learning rates prevent memorization by making it impossible for the learned score to be arbitrarily close to the empirical optimal score.

denoising score matching memorization prevention large learning rates stochastic gradient descent neural networks

2025-02-05abs ↗pdf ↗

An Adaptive and Momental Bound Method for Stochastic Learningcs.LG

Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive learning rates based on squared past gradients, e.g., AdaGrad, AdaDelta, and Adam, eases the job slightly. However, such methods have also …

2019-10-27abs ↗pdf ↗

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscs.LG

ScheduleFree+ improves large language model training without schedules or learning rates.

problem Scaling up Schedule-Free Learning to large language models.

method Learning-rate-free and schedule-free method for training large language models.

result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.

Schedule-Free Learning Large Language Models Learning Rate Model Averaging Checkpoint Merging

2026-05-18abs ↗pdf ↗

On the Origin of Implicit Regularization in Stochastic Gradient Descentcs.LG

SGD with large learning rates can achieve better test accuracy than expected.

problem SGD with large learning rates often outperforms expected convergence bounds.

method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.

result Explicitly adding an implicit regularizer to the loss improves test accuracy.

stochastic gradient descent implicit regularization large learning rates test accuracy gradient flow

2021-01-28abs ↗pdf ↗

Large Batch Training Does Not Need Warmupcs.LG

Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. However, the optimizer converges slowly at early epochs and there is a gap between large-batch deep learning optimization heuristics and theoretical underpinnings. In this paper, we propose a no…

2020-02-04abs ↗pdf ↗

How Does Learning Rate Decay Help Modern Neural Networks?cs.LG

Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (S…

2019-08-05abs ↗pdf ↗

Accelerated Large Batch Optimization of BERT Pretraining in 54 minutescs.LG

Accelerates BERT pretraining from 3 days to 54 minutes.

problem Long training time of BERT due to large mini-batch sizes.

method LANS method and learning rate scheduler for large mini-batch training.

result Achieved fastest BERT training time of 54 minutes.

BERT large batch acceleration learning rate mini-batch

2020-06-24abs ↗pdf ↗

A weak convergence approach to large deviations for stochastic approximationsmath.PR

Paper proves large deviation principle for stochastic approximations.

problem Asymptotic estimates of learning algorithm deviations.

method Weak convergence approach to large deviations.

result Identifies appropriate scaling sequence and new representation for rate function.

stochastic approximations large deviations weak convergence learning algorithms Markovian noise

2025-02-04abs ↗pdf ↗

LoRA+: Efficient Low Rank Adaptation of Large Modelscs.LG

Improved LoRA+ adapts large models more efficiently.

problem Suboptimal feature learning in LoRA for large models.

method Different learning rates for LoRA adapter matrices A and B.

result Improves performance and speed by 1-2% and up to 2X, respectively.

LoRA Low Rank Adaptation Large Models Feature Learning Efficient Adaptation

2024-02-19abs ↗pdf ↗

Large-Step Training Dynamics of a Two-Factor Linear Transformer Modelstat.ML

New insights into how large learning rates affect transformer training dynamics.

problem Understanding how large learning rates impact the training of transformer models.

method Analyzing a simplified linear transformer model with a two-factor product map.

result Large learning rates can lead to various training outcomes including cycles, chaos, or divergence.

transformer training large learning rates gradient descent chaos nonconvergence

2026-05-20abs ↗pdf ↗

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcs.LG

The study introduces anytime learning schedules for large language models without fixed horizons.

problem Training large language models without knowing the total training horizon.

method Theoretical analysis and weight averaging to create anytime learning schedules.

result Theoretical and empirical evidence shows that weight averaging with simple step sizes can achieve comparable final loss to well-tuned cosine schedules.

language models continual learning weight averaging stochastic gradient descent learning rate schedules

2026-02-03abs ↗pdf ↗

Maximal Initial Learning Rates in Deep ReLU Networksstat.ML

Maximal initial learning rate for deep ReLU networks identified.

problem Finding the optimal initial learning rate for deep neural networks.

method Simple approach to estimate maximal initial learning rate

η^{\ast}

, analyzing its behavior in constant-width fully-connected ReLU networks.

result Maximal initial learning rate

η^{\ast}

is well predicted as a power of depth × width, with specific conditions for network width and input layer training.

deep learning ReLU networks learning rate initialization maximal learning rate

2022-12-14abs ↗pdf ↗

Catapult Dynamics and Phase Transitions in Quadratic Netscs.LG

Catapult phase in neural nets shows exponential loss growth before quick decrease.

problem Understanding phase transitions in neural networks during training.

method Analyzing weight norm and loss behavior for super-critical learning rates.

result Proven existence of catapult phase in quadratic models and two-layer nets.

neural networks phase transitions learning rates weight norm ReLU activation

2023-01-18abs ↗pdf ↗

WSqD: A Horizon-Free Learning Rate Schedule for Large Model Trainingcs.LG

WSqD extends learning rate schedules for large model training without fixed horizons.

problem Fixed learning rate schedules limit training horizon extension.

method WSqD replaces constant stable phase with a shifted inverse-square-root base, retaining linear cooldown.

result WSqD achieves minimax-optimal convergence rate and horizon-independence.

learning rate schedules horizon extension stochastic convex optimization last-iterate convergence language-model pretraining

2026-07-12abs ↗pdf ↗

Where Do Large Learning Rates Lead Us?cs.LG

Large learning rates lead to optimal generalization if chosen carefully.

problem Understanding the optimal range of large learning rates for neural network training.

method Empirical study focusing on two questions: optimal initial LR range and differences between models trained with different LRs.

result Optimal initial learning rates slightly above the convergence threshold lead to optimal results after fine-tuning with a small LR or weight averaging.

neural networks learning rates generalization minima sparse features

2024-10-29abs ↗pdf ↗

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescs.LG

The paper presents a multi-power law for predicting loss curves across different learning rate schedules.

problem Understanding and optimizing the relationship between model performance and hyperparameters, especially learning rates.

method Proposes a multi-power law that combines power laws based on the sum of learning rates and additional laws for loss reduction due to decay.

result The multi-power law accurately predicts loss curves for unseen learning rate schedules and finds a schedule that outperforms cosine learning rate.

learning rate pretraining loss multi-power law loss curve prediction hyperparameter optimization

2025-03-17abs ↗pdf ↗

Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycs.LG

SGDM accelerates faster than SGD with large batch sizes and permits broader learning rates.

problem Understanding the role of momentum in SGDM and its convergence rates.

method Analysis of SGDM convergence rates under strongly convex settings, including finite-sample rates and asymptotic normality of the averaged estimator.

result SGDM converges faster than SGD with large batch sizes and permits broader learning rates.

SGDM momentum stochastic gradient descent convergence rate asymptotic normality

2023-05-28abs ↗pdf ↗

Learning-Rate-Free Learning by D-Adaptationcs.LG

D-Adaptation automatically sets optimal learning rates without manual tuning.

problem Optimizing learning rates for efficient convergence in machine learning.

method D-Adaptation, which asymptotically achieves optimal learning rates without back-tracking or additional evaluations.

result D-Adaptation automatically matches hand-tuned learning rates across diverse problems.

learning rate optimization machine learning convex functions convergence

2023-01-18abs ↗pdf ↗

Gradient-only line searches: An Alternative to Probabilistic Line Searchesstat.ML

Step sizes in neural network training are largely determined using predetermined rules such as fixed learning rates and learning rate schedules. These require user input or expensive global optimization strategies to determine their functional form and associated hyperparameters. Line searches are capable of adaptively…

2019-03-22abs ↗pdf ↗

EUR-USD Exchange Rate Forecasting Based on Information Fusion with Large Language Models and Deep Learning Methodsq-fin.CP

IUS framework predicts EUR/USD exchange rate with improved accuracy.

problem Accurate forecasting of EUR/USD exchange rate.

method Combines large language models for sentiment analysis, deep learning for forecasting, and feature selection.

result Optuna-optimized Bi-LSTM model reduces MAE and RMSE by 10.69% and 9.56% respectively.

EUR/USD exchange rate forecasting information fusion large language models deep learning

2024-08-23abs ↗pdf ↗

From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcs.LG

New learning rate approach reveals phase transitions in SGD performance.

problem Understanding feature learning dynamics in neural networks.

method Characterizing the relationship between learning rate(s) and sample complexity for gradient-based algorithms.

result Phase transition from information exponent to generative exponent regime with different learning rates.

neural networks gradient-based learning phase transitions learning rate sample complexity

2025-10-23abs ↗pdf ↗

Parsimonious Computing: A Minority Training Regime for Effective Prediction in Large Microarray Expression Data Setscs.LG

A new method reduces computational cost for gene expression inference in large microarray data sets.

problem Efficiently predicting gene expression in large datasets with limited resources.

method Adaptive Lipschitz constant inspired learning rate, random sub-sampling, and A-ReLU activation function.

result Remarkable improvement in saving computational cost while maintaining prediction accuracy.

gene expression microarray data shallow neural networks learning rate random sub-sampling

2020-05-18abs ↗pdf ↗

Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problemscs.LG

Learning rate annealing helps even in convex problems, improving generalization.

problem Improving generalization in neural networks, especially convex problems.

method Learning rate annealing schedule (large initial, then small learning rate).

result Gradient descent can reach minima with better generalization using learning rate annealing.

learning rate annealing convex problems generalization neural networks

2020-05-15abs ↗pdf ↗

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecs.LG

This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.

problem Quantifying optimal hyperparameters for large language models across scales.

method Developed three metrics to quantify hyperparameter transfer and investigated the importance of embedding layer learning rate.

result Maximal Update (μP) parameterization offers high-quality learning rate transfer compared to standard parameterization (SP).

hyperparameter transfer learning rate large language models embedding layer Maximal Update

2026-05-20abs ↗pdf ↗

SGD with a Constant Large Learning Rate Can Converge to Local Maximacs.LG

SGD with large learning rates can converge to local maxima.

problem Understanding the behavior of SGD with large learning rates.

method Constructing worst-case optimization problems.

result SGD can converge to local maxima under certain conditions.

SGD local maxima large learning rates stochastic gradient descent worst-case analysis

2021-07-25abs ↗pdf ↗

Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescs.LG

In this paper, we describe a phenomenon, which we named "super-convergence", where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-converge…

2017-08-23abs ↗pdf ↗

The QLBS Model within the presence of feedback loops through the impacts of a large traderq-fin.MF

The QLBS model is enhanced with a large trader's impact, leading to optimal hedging strategies.

problem Finding an optimal hedging strategy with low transaction costs and fair price convergence.

method Extending the QLBS model, defining a hypothetical limit order book, and using batch-mode reinforcement learning.

result Optimal hedging strategy with lower transaction costs and fair price convergence.

QLBS model large trader reinforcement learning hedging strategy fair price

2023-11-12abs ↗pdf ↗

Online Learning-guided Learning Rate Adaptation via Gradient Alignmentcs.LG

GALA adapts learning rates online by aligning gradients, improving deep learning model performance.

problem Fine-tuning learning rates for deep learning models requires extensive grid search.

method GALA dynamically adjusts learning rates by tracking gradient alignment and local curvature.

result GALA produces a flexible, adaptive learning rate schedule that increases when gradients align.

learning rate adaptation gradient alignment online learning deep learning SGD

2025-06-10abs ↗pdf ↗

On the Rate of Gaussian Approximation for Linear Regression Problemsstat.ML

Paper studies Gaussian approximation in linear regression with rates derived.

problem Gaussian approximation in online linear regression.

method Derives rates for constant learning rate settings, analyzes dependence on

d

and design matrix.

result Rate of normal approximation is

\sqrt{\log{n}/n}

for large

n

Gaussian approximation linear regression online learning rate of convergence dimensionality

2025-09-17abs ↗pdf ↗

Gradient Boosted Binary Histogram Ensemble for Large-scale Regressionstat.ML

Proposes GBBHE for efficient large-scale regression.

problem Large-scale regression problems.

method Gradient Boosting with binary histogram partition and ensemble learning.

result Improves computational efficiency and performance on large datasets.

Gradient Boosting Histogram Partition Large-scale Regression Statistical Convergence Ensemble Learning

2021-06-03abs ↗pdf ↗

The Optimization Landscape of SGD Across the Feature Learning Strengthcs.LG

Optimization of neural networks scales with γ, revealing unique loss curves and optimal learning rates.

problem Understanding the impact of feature learning strength on neural network optimization.

method Empirical investigation of neural networks with varying γ, analyzing the

γ

η

plane, and examining loss curves.

result Optimal learning rate scales non-trivially with γ, with

η^* \propto γ^2

for small γ and

η^* \propto γ^{2/L}

for large γ.

neural networks feature learning SGD optimization learning rate under-explored

2024-10-06abs ↗pdf ↗

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescs.LG

New theory shows large learning rates prevent overfitting in neural networks.

problem Generalization of two-layer ReLU neural networks in noisy regression problems.

method Gradient descent with constant learning rate converges to stable minima.

result Gradient descent with large learning rates finds smooth, sparse fits.

ReLU networks generalization minima stability large step sizes nonparametric regression

2024-06-10abs ↗pdf ↗

Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysisstat.ML

The study analyzes how machine learning classifiers' error rates decrease exponentially based on large deviations theory.

problem Understanding the convergence rate of machine learning classifiers' error probabilities.

method Large deviations theory applied to machine learning classification techniques.

result The error probability of ML classifiers converges to zero exponentially, with a rate dependent on the training set size.

Machine Learning Large Deviations Classification Error Exponential Convergence Cumulant-Generating Function

2022-07-22abs ↗pdf ↗

Agile Online Model Selection: Resolving Adaptation Lag via Safeguarded Large Learning Ratescs.LG

New algorithm reduces adaptation lag in online model selection.

problem Adaptation lag in online model selection for non-stationary environments.

method Optimistic online mirror descent with safeguarded large learning rates.

result Reduces adaptation lag from hundreds of rounds to a few rounds.

online model selection non-stationary environments agile algorithms learning rates dynamic regret

2026-05-26abs ↗pdf ↗

LipschitzLR: Using theoretically computed adaptive learning rates for fast convergencecs.LG

Optimizing deep neural networks is largely thought to be an empirical process, requiring manual tuning of several hyper-parameters, such as learning rate, weight decay, and dropout rate. Arguably, the learning rate is the most important of these to tune, and this has gained more attention in recent works. In this paper…

2019-02-20abs ↗pdf ↗