New method AdaMod stabilizes deep neural network training by limiting adaptive learning rates.
problem Adaptive learning rates can produce extremely large values at the start of training, hindering learning.
method AdaMod uses adaptive and momental upper bounds to restrict learning rates dynamically.
result AdaMod eliminates large learning rates and improves training on complex networks.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
GALA adapts learning rates online by aligning gradients, improving deep learning model performance.
problem Fine-tuning learning rates for deep learning models requires extensive grid search.
method GALA dynamically adjusts learning rates by tracking gradient alignment and local curvature.
result GALA produces a flexible, adaptive learning rate schedule that increases when gradients align.
Adaptive learning rates improve FTPL's BOBW guarantees in bandit problems.
problem Improving Follow-the-Perturbed-Leader's BOBW guarantees in bandit problems.
method Introducing surrogate probability functions to compute adaptive learning rates without exact probabilities.
result BOBW guarantees for FTPL with Pareto perturbations for any α > 1 α>1 α > 1 . POLA adapts learning rates for online time series prediction.
problem Adapting to changing data distributions in dynamic environments.
method Adaptive learning rate regulation for recurrent neural networks.
result POLA outperforms other online prediction methods in real-world datasets.
GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.
problem Fixed learning rate in AdaGrad leads to step size decrease over time.
method Introduces GradaGrad, which grows or shrinks the learning rate based on a different accumulation in the denominator.
result GradaGrad achieves similar convergence rates as AdaGrad and demonstrates non-monotone adaptation.
New adaptive learning rate for FTRL reduces regret to Θ(T^2/3).
problem Minimax regret of Θ(T^2/3) in online learning.
method Adaptive learning rate framework matching stability, penalty, and bias terms.
result Improves Best-of-Both-Worlds (BOBW) regret upper bounds.
Improved reinforcement learning with adaptive learning rates.
problem Enhancing the convergence rate of reinforcement learning algorithms.
method Two time-scale linear stochastic approximation algorithms, using Lyapunov functions and adaptive learning rates.
result Adaptive learning rate scheme significantly improves convergence rate over fixed learning rates.
CProp scales learning rate based on past gradient direction.
problem Adaptive learning rate scheduling during training.
method Gradient scaling method based on past gradient conformity.
result Significant gain in training speed on SGD and adaptive methods.
AvaGrad optimizes vision tasks by decoupling learning rate and adaptability.
problem Improving optimization methods for vision tasks.
method Derives AvaGrad, a new optimizer that decouples learning rate and adaptability.
result AvaGrad outperforms SGD on vision tasks when adaptability is properly tuned.
Local AdaAlter reduces communication in SGD with adaptive learning rates.
problem Communication overhead in distributed training.
method Novel SGD variant with adaptive learning rates and reduced communication.
result Empirically reduces communication overhead by up to 30%.
Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.
problem Analyzing risk and learning rate dynamics in high-dimensional optimization problems.
method Developed a framework to give exact expressions for risk and learning rate curves using ODEs.
result Exact expressions for risk and learning rate curves, with detailed analysis of two adaptive learning rates.
Unintended effects from scaling neural network outputs with adaptive learning rates.
problem Adaptive learning rate optimization's behavior is altered by output scaling, leading to misinterpretation.
method Presented a modified optimization algorithm to mitigate unintended effects.
result Adaptive learning rate's effectiveness is significantly impacted by output scaling, especially for small scaling factors.
Prodigy estimates learning rate without tuning, improving convergence.
problem Estimating optimal learning rate in adaptive methods.
method Prodigy modifies D-Adaptation to estimate distance to solution D D D . result Prodigy improves convergence rate by a factor of O ( log ( D / d 0 ) ) O(\sqrt{\log(D/d_0)}) O ( log ( D / d 0 ) ) . A new method to stabilize training by reducing the variance of adaptive learning rates.
problem Large variance of adaptive learning rates in the early stage of training.
method Introducing a warmup phase and a variance rectification term in RAdam.
result RAdam improves convergence and generalization in various tasks.
D-Adaptation automatically sets optimal learning rates without manual tuning.
problem Optimizing learning rates for efficient convergence in machine learning.
method D-Adaptation, which asymptotically achieves optimal learning rates without back-tracking or additional evaluations.
result D-Adaptation automatically matches hand-tuned learning rates across diverse problems.
This paper explains why Adam generalizes worse than SGD by analyzing its components.
problem Understanding why Adam generalizes worse than Stochastic Gradient Descent (SGD).
method Diffusion theoretical framework to disentangle the effects of Adaptive Learning Rate and Momentum.
result Adaptive Learning Rate helps escape saddle points but not select flat minima, while Momentum provides a drift effect to help pass through saddle points.
New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.
problem Current deep learning models lack transparency and high convergence rates.
method Reconsider neural networks as single-species population dynamics with balanced growth and harvesting rates.
result SGD with balanced growth and harvesting rates outperforms adaptive methods in all three requirements.
Study separates learning rate effects from adaptive gradient methods.
problem Understanding the impact of learning rates on neural network training.
method Introduced a 'grafting' experiment to isolate learning rate effects.
result Many existing beliefs about adaptive gradient methods may be incorrect.
DARec adapts rating patterns across domains without auxillary info.
problem Cross-domain recommendation challenges.
method Deep domain adaptation model (DARec) that transfers rating patterns.
result Best performance on public datasets.
Adaptive learning rate improves model training flexibility.
problem Limited flexibility of hand-designed learning rate schedules.
method Reinforcement learning to automatically learn adaptive learning rates.
result Auto-learned learning rate controller achieves better test results.
RAME adapts learning rates using recent first moment of gradients.
problem Training deep neural networks efficiently and adaptively.
method RAME computes individual learning rates using the most recent first moment of gradients.
result RAME outperforms SHB, Adam, and RMSprop in convergence speed and generalization performance.
We present the first adaptive strategy for active learning in the setting of classification with smooth decision boundary. The problem of adaptivity (to unknown distributional parameters) has remained opened since the seminal work of Castro and Nowak (2007), which first established (active learning) rates for this sett…
New adaptive methods improve on generalization and speed.
problem Adaptive methods like Adam and RMSprop generalize poorly and can lead to unstable learning rates.
method AdaBound and AMSBound use dynamic learning rate bounds to stabilize and smooth transitions.
result New methods eliminate generalization gap and maintain higher learning speed early in training.
New algorithm reduces adaptation lag in online model selection.
problem Adaptation lag in online model selection for non-stationary environments.
method Optimistic online mirror descent with safeguarded large learning rates.
result Reduces adaptation lag from hundreds of rounds to a few rounds.
A new method calculates adaptive learning rates for faster neural network training.
problem Finding the optimal learning rate for deep neural networks.
method Derives a theoretical framework to compute learning rates dynamically based on the Lipschitz constant of the loss function, and extends this to other optimization algorithms.
result Commonly used learning rates are significantly smaller than the ideal value, leading to faster convergence.
Paper analyzes convergence rates for multi-agent learning in games.
problem Convergence rates for multi-agent learning in games.
method Characterizes finite-time convergence rates for joint OGD learning on λ λ λ -cocoercive games and develops adaptive algorithms. result Adaptive algorithms achieve same convergence rates as non-adaptive counterparts.
Automates learning rate tuning in machine learning.
problem Difficulty in tuning the learning rate of stochastic gradient methods.
method Automates the learning rate tuning by using a statistical test to determine when to decrease the learning rate.
result Statistical adaptive stochastic approximation (SASA) method can automatically find good learning rate schedules and match hand-tuned methods.
New adaptive learning rate improves FTRL's adaptivity to sparsity, game-dependency, and best-of-both-worlds.
problem Improving adaptivity in sequential decision-making problems.
method Developed a stability-penalty-adaptive (SPA) learning rate for FTRL.
result First BOBW algorithm with sparsity-dependent bound.
Improved LoRA+ adapts large models more efficiently.
problem Suboptimal feature learning in LoRA for large models.
method Different learning rates for LoRA adapter matrices A and B.
result Improves performance and speed by 1-2% and up to 2X, respectively.
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
Adaptive algorithm improves convergence rate of Langevin dynamics.
problem Improving convergence rate of Langevin dynamics.
method Adaptive non-reversible stochastic gradient Langevin dynamics algorithm.
result Improved convergence rate of the algorithm.
New guarantees for ERM with adaptively collected data.
problem Failure of ERM guarantees with adaptively collected data.
method Importance sampling weighted ERM algorithm with maximal inequality.
result First generalization guarantees and fast convergence rates for adaptively collected data.
AdaScale SGD adapts learning rates for large-batch training efficiently.
problem Adapting learning rates for large-batch training to balance speed-ups and model quality.
method Adaptive learning rate adaptation based on gradient variance.
result AdaScale achieves reliable speed-ups for a wide range of batch sizes without degrading model quality.
MaxVA improves Adam's step sizes by maximizing gradient variance.
problem Unstable or extreme adaptive learning rates in Adam.
method Maximizing the variance of gradient coordinates in Adam's running mean of squared gradients.
result Faster adaptation and more desirable convergence behaviors than Adam.
MetaGrad adapts multiple learning rates for faster online optimization.
problem Online convex optimization with varying function types.
method MetaGrad uses multiple adaptive learning rates based on empirical performance.
result MetaGrad achieves faster rates for various function types.
Auto-Ensemble automates deep learning model ensembling with adaptive learning rate scheduling.
problem Difficulty in collecting diverse and accurate deep learning models through single training.
method Auto-Ensemble collects model checkpoints and uses adaptive learning rate scheduling to ensemble them.
result Ensembled models converge to various local optima, improving performance on few-shot learning.
Adaptive weights improve physics-informed neural networks and deep operator networks.
problem Training physics-informed neural networks and deep operator networks can be challenging, leading to unsatisfactory accuracy and efficiency.
method Proposes a pointwise adaptive weighting method that balances the residual decay rate across different training points.
result Our proposed approach of balanced residual decay rates offers advantages including bounded weights, high prediction accuracy, fast convergence rate, low training uncertainty, low computational cost, and ease of hyperparameter tuning.
Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…
Adaptive online learning algorithms without manual tuning of a Lipschitz hyperparameter.
problem Designing adaptive online learning algorithms that require minimal user input and automatically adjust hyperparameters.
method Developing new versions of MetaGrad and Squint algorithms that dynamically update learning rates and adapt to the optimal Lipschitz hyperparameter.
result Automatic adaptation of the Lipschitz hyperparameter, improving performance and efficiency of online learning algorithms.
Paper studies transfer learning for nonparametric classification, establishing rates and proposing adaptive classifiers.
problem Transfer learning in nonparametric classification under different distributions.
method Established minimax rates and proposed adaptive classifiers based on weighted K-NN approach.
result Data-driven adaptive classifier achieves near-optimal rates over various parameter spaces.
FedDuA adapts global learning rate for federated learning.
problem Slow convergence in federated learning due to dataset and parameter space heterogeneity.
method FedDuA uses mirror descent to adaptively select global learning rate based on inter-client and coordinate-wise heterogeneity.
result FedDuA achieves minimax optimal convergence for convex objectives and outperforms baselines in various settings.
New adaptive SGD algorithms for federated learning over physical channels.
problem Reducing communication cost in federated learning over physical channels.
method Proposed adaptive federated SGD algorithms considering channel noise and hardware constraints.
result Demonstrated convergence rates adaptive to stochastic gradient noise level.
Wireless systems perform rate adaptation to transmit at highest possible instantaneous rates. Rate adaptation has been increasingly granular over generations of wireless systems. The base-station uses SINR and packet decode feedback called acknowledgement/no acknowledgement (ACK/NACK) to perform rate adaptation. SINR i…
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
Paper analyzes high probability convergence of adaptive SGD with momentum.
problem Theoretical understanding of adaptive SGD with momentum in nonconvex settings is incomplete.
method High probability analysis under weak assumptions.
result First high probability convergence proof for gradients to zero in Delayed AdaGrad with momentum.
Adaptive-SGD method optimizes machine learning training with dynamic batch and step sizes.
problem Optimizing machine learning training with adaptive batch and step sizes.
method Adaptive-SGD method that dynamically adjusts batch size and step size based on local curvature and probability of descent directions.
result Adaptive-SGD achieves global linear convergence on self-concordant functions and compares favorably to fine-tuned methods.
SALSA automatically adjusts learning rates in stochastic gradient methods.
problem Automatic adjustment of learning rates in stochastic gradient methods.
method SALSA uses a line-search procedure to gradually increase the learning rate, then a statistical test to decrease it.
result SALSA matches the performance of best hand-tuned learning rate schedules in deep learning tasks.