arXiv research

LALR adapts learning rate for faster convergence in regression and neural nets.

GALA adapts learning rates online by aligning gradients, improving deep learning model performance.

Adaptive learning rates improve FTPL's BOBW guarantees in bandit problems.

Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive learning rates based on squared past gradients, e.g., AdaGrad, AdaDelta, and Adam, eases the job slightly. However, such methods have also …

POLA adapts learning rates for online time series prediction.

GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.

Study on distributed nonparametric function estimation with optimal rate and cost of adaptation.

New adaptive learning rate for FTRL reduces regret to Θ(T^2/3).

Unintended effects from scaling neural network outputs with adaptive learning rates.

This paper explains why Adam generalizes worse than SGD by analyzing its components.

Prodigy estimates learning rate without tuning, improving convergence.

Adapts to estimate functions from noisy ERT data.

Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.

Adaptive weights improve physics-informed neural networks and deep operator networks.

We present the first adaptive strategy for active learning in the setting of classification with smooth decision boundary. The problem of adaptivity (to unknown distributional parameters) has remained opened since the seminal work of Castro and Nowak (2007), which first established (active learning) rates for this sett…

Study shows rates for Laplacian-eigenmap methods in nonparametric regression.

Adaptive algorithm improves convergence rate of Langevin dynamics.

Study separates learning rate effects from adaptive gradient methods.

Rate-In dynamically adjusts dropout rates during inference to improve uncertainty estimation in neural networks.

New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.

New protocols show 1-bit mean estimation can be order-optimal without interaction.

D-Adaptation automatically sets optimal learning rates without manual tuning.

New algorithm reduces adaptation lag in online model selection.

Wireless systems perform rate adaptation to transmit at highest possible instantaneous rates. Rate adaptation has been increasingly granular over generations of wireless systems. The base-station uses SINR and packet decode feedback called acknowledgement/no acknowledgement (ACK/NACK) to perform rate adaptation. SINR i…

Study on signal detection in sparse additive models with nonasymptotic minimax rates.

Most optimizers including stochastic gradient descent (SGD) and its adaptive gradient derivatives face the same problem where an effective learning rate during the training is vastly different. A learning rate scheduling, mostly tuned by hand, is usually employed in practice. In this paper, we propose CProp, a gradient…

Improved SGD methods converge faster for nonconvex optimization.

Research shows minimal communication limits adaptive function estimation rates.

Derives effective continuous dynamics for adaptive SGD methods.

New adaptive learning rate improves FTRL's adaptivity to sparsity, game-dependency, and best-of-both-worlds.

MaxVA improves Adam's step sizes by maximizing gradient variance.

Distributed sensors compress and send features to a fusion center for linear regression.

AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.

Improved LoRA+ adapts large models more efficiently.

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the ad…

New algorithm for adaptive experimental design in scientific settings.

Paper analyzes convergence rates for multi-agent learning in games.

New guarantees for ERM with adaptively collected data.

AdaScale SGD adapts learning rates for large-batch training efficiently.

Adaptive momentum method solves non-convex min-max problems.

Adapts SGD to noise and problem specifics for faster convergence.

Adaptive importance samplers are adaptive Monte Carlo algorithms to estimate expectations with respect to some target distribution which \textit{adapt} themselves to obtain better estimators over a sequence of iterations. Although it is straightforward to show that they have the same $\mathcal{O}(1/\sqrt{N})$ convergen…

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the gene…

Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…

MetaGrad adapts multiple learning rates for faster online optimization.

New algorithms optimize private convex optimization with faster rates for functions with κ-growth.

From a simplified analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned. We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search. I…

Paper analyzes high probability convergence of adaptive SGD with momentum.