The paper analyzes how learning rate affects SGD and provides insights into optimal rates.
problem Understanding the impact of learning rate on stochastic gradient descent.
method Developed a learning-rate-dependent stochastic differential equation (lr-dependent SDE) to analyze SGD.
result Established a linear rate of convergence for SGD and found the optimal linear rate by analyzing the spectrum of the Witten-Laplacian.
GALA adapts learning rates online by aligning gradients, improving deep learning model performance.
problem Fine-tuning learning rates for deep learning models requires extensive grid search.
method GALA dynamically adjusts learning rates by tracking gradient alignment and local curvature.
result GALA produces a flexible, adaptive learning rate schedule that increases when gradients align.
Cyclical learning rate improves neural machine translation performance.
problem Optimizing learning rate for neural machine translation.
method Applied cyclical learning rate to transformer-based neural networks.
result Cyclical learning rate significantly impacts neural machine translation performance.
Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is anne…
Cyclical learning rates improve DRL performance without manual tuning.
problem Manual hyperparameter tuning in DRL is time-consuming and error-prone.
method Proposes cyclical learning rates for DRL problems.
result Cyclical learning achieves similar or better results than fixed learning rates.
SALR improves deep learning generalization by dynamically adjusting learning rates.
problem Improving generalization in deep learning models.
method Sharpness-aware learning rate scheduling based on local loss function sharpness.
result SALR drives solutions to flatter regions, improving generalization and convergence.
Learning rate annealing helps even in convex problems, improving generalization.
problem Improving generalization in neural networks, especially convex problems.
method Learning rate annealing schedule (large initial, then small learning rate).
result Gradient descent can reach minima with better generalization using learning rate annealing.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.
problem Analyzing risk and learning rate dynamics in high-dimensional optimization problems.
method Developed a framework to give exact expressions for risk and learning rate curves using ODEs.
result Exact expressions for risk and learning rate curves, with detailed analysis of two adaptive learning rates.
In this paper, we give a new sharp generalization bound of lp-MKL which is a generalized framework of multiple kernel learning (MKL) and imposes lp-mixed-norm regularization instead of l1-mixed-norm regularization. We utilize localization techniques to obtain the sharp learning rate. The bound is characterized by the d…
D-Adaptation automatically sets optimal learning rates without manual tuning.
problem Optimizing learning rates for efficient convergence in machine learning.
method D-Adaptation, which asymptotically achieves optimal learning rates without back-tracking or additional evaluations.
result D-Adaptation automatically matches hand-tuned learning rates across diverse problems.
Local update methods' performance depends on learning rates, affecting convergence rates and alignment with true loss.
problem The performance of local update methods in federated learning and meta-learning is sensitive to learning rates.
method Proved that local update methods perform SGD on a surrogate loss function, characterized the surrogate loss, and derived convergence rates.
result Proper learning rate tuning is crucial for near-optimal behavior in communication-limited settings.
Proof of learning rate transfer in MLPs with μP parameterization.
problem Understanding and optimizing learning rates in neural networks with different parameterizations.
method Theoretical analysis and empirical validation of learning rate transfer in MLPs with μP, SP, and NTP parameterizations. result The optimal learning rate converges to a non-zero constant as width goes to infinity under μP, explaining learning rate transfer. Large learning rates enhance model robustness and compressibility.
problem Achieving robustness and resource-efficiency in machine learning models.
method Identifying and utilizing large learning rates as a facilitator for robustness and compressibility.
result Large learning rates produce desirable representation properties and compare favorably to other methods.
The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confirm their predictions empirically in practical deep learning settings. The networks exhibit sharply distinct behaviors at small and large lea…
Random learning rate improves neural network training without extra cost.
problem Optimizing neural networks with stochastic gradient descent.
method Uniformly-distributed random learning rate in SGD.
result Random learning rate yields better regularization without extra computational cost.
Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…
A novel approach models rating transitions using Lie groups and Deep Learning.
problem Modeling rating transitions with geometric properties and stochastic processes.
method Introducing Itô-SDEs on Lie groups, using TimeGAN for calibration, and examining rating matrix properties.
result The geometric approach using Lie groups and Deep Learning generates a good fit for rating transitions.
Paper improves learning rates for SGD and NAG.
problem Generalization performance of stochastic optimization algorithms.
method Establishes new learning rates for SGD and NAG.
result Improved guarantees in some settings or comparable rates under weaker assumptions.
Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive learning rates based on squared past gradients, e.g., AdaGrad, AdaDelta, and Adam, eases the job slightly. However, such methods have also …
Maximal initial learning rate for deep ReLU networks identified.
problem Finding the optimal initial learning rate for deep neural networks.
method Simple approach to estimate maximal initial learning rate η∗, analyzing its behavior in constant-width fully-connected ReLU networks. result Maximal initial learning rate η∗ is well predicted as a power of depth × width, with specific conditions for network width and input layer training. AutoSGD automatically adjusts learning rates for SGD.
problem Inefficient manual learning rate selection for SGD.
method AutoSGD automatically adjusts learning rates based on iteration.
result Empirical results show strong performance across various optimization tasks.
Optimal learning rate schedules for SGD in changing data distributions.
problem Minimizing regret in online learning with changing data distributions.
method Characterized optimal schedules for linear regression, proposed schedules for general convex and non-convex losses, and defined a notion of regret for non-convex losses.
result Upper and lower bounds for regret with constants for convex losses, and an upper bound on total expected regret for non-convex losses.
Learning-rate schedules for large models match optimization theory closely, leading to better training.
problem Improving training of large models with optimal learning rates.
method Used a bound from non-smooth convex optimization theory to match learning-rate schedules with practical benefits.
result Extending the learning-rate schedule with optimal learning-rate and transferring it across schedules improves model training.
AutoGD automatically adjusts learning rates for gradient descent.
problem Optimizing learning rates for gradient descent methods.
method AutoGD automatically adjusts learning rates based on iteration.
result AutoGD can recover the optimal rate of GD for a broad class of functions.
Adaptive learning rates improve FTPL's BOBW guarantees in bandit problems.
problem Improving Follow-the-Perturbed-Leader's BOBW guarantees in bandit problems.
method Introducing surrogate probability functions to compute adaptive learning rates without exact probabilities.
result BOBW guarantees for FTPL with Pareto perturbations for any α>1. Paper establishes convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
problem Analyzing convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
method Novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing length.
result First almost sure convergence rate and maximal concentration bound with exponential tails for contractive stochastic approximation algorithms with Markovian noise.
We improve private training accuracy with learning rate schedules and matrix factorizations.
problem Private training with learning rate schedules and correlated noise.
method General upper and lower bounds for learning rate schedules, memory-efficient constructions, and schedule-aware factorizations.
result Schedule-aware factorizations improve accuracy in private training.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Develops methods to measure and set function-space learning rates in neural networks.
problem Measuring and optimizing changes in neural network output functions.
method Efficient methods to measure and set function-space learning rates, requiring minimal computational overhead.
result Demonstrates FLeRM (Function-space Learning Rate Matching) for hyperparameter transfer across model scales.
The learning rate is one of the most important hyper-parameters for model training and generalization. However, current hand-designed parametric learning rate schedules offer limited flexibility and the predefined schedule may not match the training dynamics of high dimensional and non-convex optimization problems. In …
New adaptive learning rate for FTRL reduces regret to Θ(T^2/3).
problem Minimax regret of Θ(T^2/3) in online learning.
method Adaptive learning rate framework matching stability, penalty, and bias terms.
result Improves Best-of-Both-Worlds (BOBW) regret upper bounds.
A new method automatically and dynamically sets learning rates in deep learning.
problem Determining the appropriate learning rate in deep learning tasks is challenging and often subjective.
method Local Quadratic Approximation (LQA) to automatically and dynamically set learning rates.
result The proposed method leads to nearly optimal learning rates in a computationally efficient way.
WSqD extends learning rate schedules for large model training without fixed horizons.
problem Fixed learning rate schedules limit training horizon extension.
method WSqD replaces constant stable phase with a shifted inverse-square-root base, retaining linear cooldown.
result WSqD achieves minimax-optimal convergence rate and horizon-independence.
Large learning rates cause oscillations in NN weights that improve generalization.
problem Improving generalization of neural networks trained with large learning rates.
method Theoretical analysis and feature-noise data generation model.
result Oscillating SGD with large learning rates benefits NN generalization by effectively learning weak features.
Study shows deep linear networks can converge to flatter minima at large learning rates.
problem Understanding the implicit bias of deep linear networks at large learning rates.
method Characterization of deep linear networks for binary classification using logistic loss in the large learning rate regime.
result Gradient descent iterates converge to a flatter minimum in the catapult phase for certain data separation conditions.
This paper demonstrates dynamic hyper-parameter setting, for deep neural network training, using Mutual Information (MI). The specific hyper-parameter studied in this paper is the learning rate. MI between the output layer and true outcomes is used to dynamically set the learning rate of the network through the trainin…
nGPT learns to transfer learning rates across model dimensions and token horizons.
problem nGPT does not transfer learning rates across model size and token horizon.
method Combining numerical experiments with alignment exponents, a novel nGPT parameterization νGPT is developed.
result νGPT exhibits learning rate transfer across width, depth, and token horizon.
The paper presents a multi-power law for predicting loss curves across different learning rate schedules.
problem Understanding and optimizing the relationship between model performance and hyperparameters, especially learning rates.
method Proposes a multi-power law that combines power laws based on the sum of learning rates and additional laws for loss reduction due to decay.
result The multi-power law accurately predicts loss curves for unseen learning rate schedules and finds a schedule that outperforms cosine learning rate.
Fractal learning rate schedules accelerate vanilla gradient descent.
problem Difficulty in tuning learning rates in iterative optimization.
method Introduce Chebyshev learning rate schedule for gradient descent.
result Locally unstable updates can lead to convergence in deep learning.
We consider the problem of learning a non-negative linear classifier with a 1-norm of at most k, and a fixed threshold, under the hinge-loss. This problem generalizes the problem of learning a k-monotone disjunction. We prove that we can learn efficiently in this setting, at a rate which is linear in both k and…
New learning rate approach reveals phase transitions in SGD performance.
problem Understanding feature learning dynamics in neural networks.
method Characterizing the relationship between learning rate(s) and sample complexity for gradient-based algorithms.
result Phase transition from information exponent to generative exponent regime with different learning rates.
Optimal learning rate schedules derived for various tasks.
problem Inadequate learning rate schedules in practice compared to theory.
method Refined analysis of learning rate schedules for optimization algorithms.
result Derives new problem-adaptive learning rate schedules.
Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.
problem Inefficiencies in ad-hoc learning rate tuning for optimization algorithms.
method Developed AdaGrad++ and Adam++ without predefined learning rates and proved their convergence.
result AdaGrad++ and Adam++ achieve comparable convergence rates to AdaGrad and Adam respectively.
New coin sampling method for Bayesian inference without learning rates.
problem Scalable Bayesian inference with learning rate tuning issues.
method Coin sampling for gradient-based Bayesian inference.
result Comparable performance to other ParVI algorithms without learning rate tuning.
Optimizing deep neural networks is largely thought to be an empirical process, requiring manual tuning of several hyper-parameters, such as learning rate, weight decay, and dropout rate. Arguably, the learning rate is the most important of these to tune, and this has gained more attention in recent works. In this paper…
Learning rate annealing improves robustness in stochastic optimization.
problem Tuning learning rates in large-scale models is costly and prone to errors.
method We analyze and demonstrate the benefits of learning rate annealing schemes.
result Stochastic gradient descent with annealed schedules converges more robustly to the optimal solution.
GOLS-I automatically determines learning rates for various neural network training algorithms.
problem Adapting learning rates in stochastic training algorithms for neural networks.
method Gradient-Only Line Search (GOLS-I) for automatically setting learning rates.
result GOLS-I learning rate schedules are competitive with manually tuned rates across multiple algorithms, architectures, datasets, and loss functions.