New method AdaMod stabilizes deep neural network training by limiting adaptive learning rates.
problem Adaptive learning rates can produce extremely large values at the start of training, hindering learning.
method AdaMod uses adaptive and momental upper bounds to restrict learning rates dynamically.
result AdaMod eliminates large learning rates and improves training on complex networks.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
GALA adapts learning rates online by aligning gradients, improving deep learning model performance.
problem Fine-tuning learning rates for deep learning models requires extensive grid search.
method GALA dynamically adjusts learning rates by tracking gradient alignment and local curvature.
result GALA produces a flexible, adaptive learning rate schedule that increases when gradients align.
Adaptive learning rates improve FTPL's BOBW guarantees in bandit problems.
problem Improving Follow-the-Perturbed-Leader's BOBW guarantees in bandit problems.
method Introducing surrogate probability functions to compute adaptive learning rates without exact probabilities.
result BOBW guarantees for FTPL with Pareto perturbations for any α > 1 α>1 α > 1 . Reinforcement learning improves wireless systems' rate adaptation.
problem Optimizing rate adaptation in 4G/5G systems using ACK/NACK feedback.
method Formulated as a Multi-Armed Bandit problem, proposed binary search algorithm with PAC guarantees.
result Achieved PAC solution for OLLA with binary search, outperforming UCB methods.
POLA adapts learning rates for online time series prediction.
problem Adapting to changing data distributions in dynamic environments.
method Adaptive learning rate regulation for recurrent neural networks.
result POLA outperforms other online prediction methods in real-world datasets.
GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.
problem Fixed learning rate in AdaGrad leads to step size decrease over time.
method Introduces GradaGrad, which grows or shrinks the learning rate based on a different accumulation in the denominator.
result GradaGrad achieves similar convergence rates as AdaGrad and demonstrates non-monotone adaptation.
AvaGrad optimizes vision tasks by decoupling learning rate and adaptability.
problem Improving optimization methods for vision tasks.
method Derives AvaGrad, a new optimizer that decouples learning rate and adaptability.
result AvaGrad outperforms SGD on vision tasks when adaptability is properly tuned.
Study on distributed nonparametric function estimation with optimal rate and cost of adaptation.
problem Optimal rate of convergence and cost of adaptation in distributed nonparametric function estimation.
method Distributed minimax estimation and adaptive estimation under communication constraints for Gaussian sequence model and white noise model.
result Established minimax rate of convergence and exact communication cost for adaptation.
New adaptive learning rate for FTRL reduces regret to Θ(T^2/3).
problem Minimax regret of Θ(T^2/3) in online learning.
method Adaptive learning rate framework matching stability, penalty, and bias terms.
result Improves Best-of-Both-Worlds (BOBW) regret upper bounds.
Local AdaAlter reduces communication in SGD with adaptive learning rates.
problem Communication overhead in distributed training.
method Novel SGD variant with adaptive learning rates and reduced communication.
result Empirically reduces communication overhead by up to 30%.
DARec adapts rating patterns across domains without auxillary info.
problem Cross-domain recommendation challenges.
method Deep domain adaptation model (DARec) that transfers rating patterns.
result Best performance on public datasets.
CProp scales learning rate based on past gradient direction.
problem Adaptive learning rate scheduling during training.
method Gradient scaling method based on past gradient conformity.
result Significant gain in training speed on SGD and adaptive methods.
Unintended effects from scaling neural network outputs with adaptive learning rates.
problem Adaptive learning rate optimization's behavior is altered by output scaling, leading to misinterpretation.
method Presented a modified optimization algorithm to mitigate unintended effects.
result Adaptive learning rate's effectiveness is significantly impacted by output scaling, especially for small scaling factors.
This paper explains why Adam generalizes worse than SGD by analyzing its components.
problem Understanding why Adam generalizes worse than Stochastic Gradient Descent (SGD).
method Diffusion theoretical framework to disentangle the effects of Adaptive Learning Rate and Momentum.
result Adaptive Learning Rate helps escape saddle points but not select flat minima, while Momentum provides a drift effect to help pass through saddle points.
Adaptive learning rate improves gradient descent for deep models.
problem Difficulty in tuning fixed learning rates for deep models.
method Proposes adaptive learning rates using either first-order or second-order methods.
result Optimized gradient descent leads to better convergence for machine learning algorithms.
Prodigy estimates learning rate without tuning, improving convergence.
problem Estimating optimal learning rate in adaptive methods.
method Prodigy modifies D-Adaptation to estimate distance to solution D D D . result Prodigy improves convergence rate by a factor of O ( log ( D / d 0 ) ) O(\sqrt{\log(D/d_0)}) O ( log ( D / d 0 ) ) . Adapts to estimate functions from noisy ERT data.
problem Estimating functions from noisy Exponential Radon Transform data.
method Locally adaptive kernel type estimator for functions of varying smoothness.
result Achieves minimax optimal rate up to a log(n) factor for Sobolev functions.
Improved reinforcement learning with adaptive learning rates.
problem Enhancing the convergence rate of reinforcement learning algorithms.
method Two time-scale linear stochastic approximation algorithms, using Lyapunov functions and adaptive learning rates.
result Adaptive learning rate scheme significantly improves convergence rate over fixed learning rates.
A new method to stabilize training by reducing the variance of adaptive learning rates.
problem Large variance of adaptive learning rates in the early stage of training.
method Introducing a warmup phase and a variance rectification term in RAdam.
result RAdam improves convergence and generalization in various tasks.
Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.
problem Analyzing risk and learning rate dynamics in high-dimensional optimization problems.
method Developed a framework to give exact expressions for risk and learning rate curves using ODEs.
result Exact expressions for risk and learning rate curves, with detailed analysis of two adaptive learning rates.
Adaptive weights improve physics-informed neural networks and deep operator networks.
problem Training physics-informed neural networks and deep operator networks can be challenging, leading to unsatisfactory accuracy and efficiency.
method Proposes a pointwise adaptive weighting method that balances the residual decay rate across different training points.
result Our proposed approach of balanced residual decay rates offers advantages including bounded weights, high prediction accuracy, fast convergence rate, low training uncertainty, low computational cost, and ease of hyperparameter tuning.
New adaptive strategy for active learning with smooth boundaries.
problem Adaptive active learning in multivariate classification with unknown distributional parameters.
method Combining insights from recent works, reduction to univariate-adaptive strategies.
result Near-optimal rates achieved without prior knowledge of distributional parameters.
Study shows rates for Laplacian-eigenmap methods in nonparametric regression.
problem Minimizing error in nonparametric regression using Laplacian-eigenmap.
method Adaptive and non-adaptive minimax rates using Sobolev space constraints.
result Extends minimax rates to various weighted Laplacian matrices.
Adaptive algorithm improves convergence rate of Langevin dynamics.
problem Improving convergence rate of Langevin dynamics.
method Adaptive non-reversible stochastic gradient Langevin dynamics algorithm.
result Improved convergence rate of the algorithm.
Study separates learning rate effects from adaptive gradient methods.
problem Understanding the impact of learning rates on neural network training.
method Introduced a 'grafting' experiment to isolate learning rate effects.
result Many existing beliefs about adaptive gradient methods may be incorrect.
Rate-In dynamically adjusts dropout rates during inference to improve uncertainty estimation in neural networks.
problem Static dropout rates lead to suboptimal uncertainty estimates in neural networks.
method Rate-In dynamically adjusts dropout rates using information-theoretic principles.
result Rate-In improves calibration and sharpens uncertainty estimates compared to fixed or heuristic dropout rates.
New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.
problem Current deep learning models lack transparency and high convergence rates.
method Reconsider neural networks as single-species population dynamics with balanced growth and harvesting rates.
result SGD with balanced growth and harvesting rates outperforms adaptive methods in all three requirements.
New protocols show 1-bit mean estimation can be order-optimal without interaction.
problem Can 1-bit mean estimation be optimal without interaction?
method Adaptive and non-adaptive threshold and interval queries, with one adaptive transition.
result Arbitrary non-adaptive quantizers can match the adaptive rate, suggesting interaction is not necessary.
D-Adaptation automatically sets optimal learning rates without manual tuning.
problem Optimizing learning rates for efficient convergence in machine learning.
method D-Adaptation, which asymptotically achieves optimal learning rates without back-tracking or additional evaluations.
result D-Adaptation automatically matches hand-tuned learning rates across diverse problems.
New algorithm reduces adaptation lag in online model selection.
problem Adaptation lag in online model selection for non-stationary environments.
method Optimistic online mirror descent with safeguarded large learning rates.
result Reduces adaptation lag from hundreds of rounds to a few rounds.
CoRAS adapts image acquisition rates for accurate reconstruction.
problem Determining when enough measurements are collected for accurate image reconstruction.
method Adaptive acquisition rate selection based on reconstruction error probability.
result CoRAS achieves target stopping-time coverage with fewer measurements.
RAME adapts learning rates using recent first moment of gradients.
problem Training deep neural networks efficiently and adaptively.
method RAME computes individual learning rates using the most recent first moment of gradients.
result RAME outperforms SHB, Adam, and RMSprop in convergence speed and generalization performance.
Study on signal detection in sparse additive models with nonasymptotic minimax rates.
problem Signal detection in sparse additive models.
method Nonasymptotic minimax analysis of signal detection in sparse additive models.
result Established minimax separation rate for signal detection.
New optimised adaptive importance samplers converge faster than standard methods.
problem Improving Monte Carlo estimators for target distributions.
method Optimised adaptive importance samplers using convex optimisation of χ 2 χ^2 χ 2 -divergence. result Convergence rate of O ( 1 / N ) \mathcal{O}(1/\sqrt{N}) O ( 1/ N ) for optimised samplers, with explicit iteration and sample dependence. Improved SGD methods converge faster for nonconvex optimization.
problem Nonconvex optimization challenges in machine learning.
method Adaptive SGD with line-search and Polyak stepsizes.
result Unified convergence rates for various nonconvex functions.
Adaptive learning rate improves SGD performance in federated and private learning.
problem Improving SGD performance in federated and private learning.
method Adaptively tunes learning rate using extrapolation of gradient flow.
result Adaptive algorithm outperforms manually tuned optimizers in federated and private learning.
Research shows minimal communication limits adaptive function estimation rates.
problem Adaptive estimation of a smooth function under minimal communication constraints.
method Investigates the L ∞ L_\infty L ∞ -risk and L 2 L_2 L 2 -risk under different numbers of servers. result For L ∞ L_\infty L ∞ -risk, optimal rates cannot be achieved under minimal communication. For L 2 L_2 L 2 -risk, adaptivity is possible but depends on server number and sample size. Improved speech recognition with cumulative adaptation methods.
problem Robust speech recognition in varying environments and speakers.
method Used a bidirectional LSTM neural network and i-vectors for adaptation.
result Achieved 13% relative improvement in word error rate.
New adaptive methods improve on generalization and speed.
problem Adaptive methods like Adam and RMSprop generalize poorly and can lead to unstable learning rates.
method AdaBound and AMSBound use dynamic learning rate bounds to stabilize and smooth transitions.
result New methods eliminate generalization gap and maintain higher learning speed early in training.
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
New adaptive learning rate improves FTRL's adaptivity to sparsity, game-dependency, and best-of-both-worlds.
problem Improving adaptivity in sequential decision-making problems.
method Developed a stability-penalty-adaptive (SPA) learning rate for FTRL.
result First BOBW algorithm with sparsity-dependent bound.
Adaptive learning rate improves model training flexibility.
problem Limited flexibility of hand-designed learning rate schedules.
method Reinforcement learning to automatically learn adaptive learning rates.
result Auto-learned learning rate controller achieves better test results.
MaxVA improves Adam's step sizes by maximizing gradient variance.
problem Unstable or extreme adaptive learning rates in Adam.
method Maximizing the variance of gradient coordinates in Adam's running mean of squared gradients.
result Faster adaptation and more desirable convergence behaviors than Adam.
Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.
problem Optimal convergence rates for non-convex, smooth objectives with adaptive step sizes.
method Adaptive SGD (AdaGrad-Norm) with self-tuning step sizes, analyzing under unbounded gradients and affine variance scaling.
result AdaGrad-Norm achieves order optimal convergence rate of $\mathcal{O}\left(\frac{\mathrm{poly}\log(T)}{\sqrt{T}}
ight)$ under optimal assumptions.
Improved LoRA+ adapts large models more efficiently.
problem Suboptimal feature learning in LoRA for large models.
method Different learning rates for LoRA adapter matrices A and B.
result Improves performance and speed by 1-2% and up to 2X, respectively.
A new method calculates adaptive learning rates for faster neural network training.
problem Finding the optimal learning rate for deep neural networks.
method Derives a theoretical framework to compute learning rates dynamically based on the Lipschitz constant of the loss function, and extends this to other optimization algorithms.
result Commonly used learning rates are significantly smaller than the ideal value, leading to faster convergence.