arXiv research

LALR adapts learning rate for faster convergence in regression and neural nets.

GALA adapts learning rates online by aligning gradients, improving deep learning model performance.

Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive learning rates based on squared past gradients, e.g., AdaGrad, AdaDelta, and Adam, eases the job slightly. However, such methods have also …

Adaptive learning rates improve FTPL's BOBW guarantees in bandit problems.

POLA adapts learning rates for online time series prediction.

GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.

New adaptive learning rate for FTRL reduces regret to Θ(T^2/3).

Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.

Unintended effects from scaling neural network outputs with adaptive learning rates.

Prodigy estimates learning rate without tuning, improving convergence.

D-Adaptation automatically sets optimal learning rates without manual tuning.

This paper explains why Adam generalizes worse than SGD by analyzing its components.

New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.

We present the first adaptive strategy for active learning in the setting of classification with smooth decision boundary. The problem of adaptivity (to unknown distributional parameters) has remained opened since the seminal work of Castro and Nowak (2007), which first established (active learning) rates for this sett…

Most optimizers including stochastic gradient descent (SGD) and its adaptive gradient derivatives face the same problem where an effective learning rate during the training is vastly different. A learning rate scheduling, mostly tuned by hand, is usually employed in practice. In this paper, we propose CProp, a gradient…

New algorithm reduces adaptation lag in online model selection.

New adaptive learning rate improves FTRL's adaptivity to sparsity, game-dependency, and best-of-both-worlds.

Improved LoRA+ adapts large models more efficiently.

Derives effective continuous dynamics for adaptive SGD methods.

Adaptive algorithm improves convergence rate of Langevin dynamics.

New guarantees for ERM with adaptively collected data.

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the ad…

AdaScale SGD adapts learning rates for large-batch training efficiently.

MaxVA improves Adam's step sizes by maximizing gradient variance.

MetaGrad adapts multiple learning rates for faster online optimization.

Auto-Ensemble automates deep learning model ensembling with adaptive learning rate scheduling.

Adaptive weights improve physics-informed neural networks and deep operator networks.

Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…

FedDuA adapts global learning rate for federated learning.

New adaptive SGD algorithms for federated learning over physical channels.

Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …

Wireless systems perform rate adaptation to transmit at highest possible instantaneous rates. Rate adaptation has been increasingly granular over generations of wireless systems. The base-station uses SINR and packet decode feedback called acknowledgement/no acknowledgement (ACK/NACK) to perform rate adaptation. SINR i…

Paper analyzes high probability convergence of adaptive SGD with momentum.

The learning rate is one of the most important hyper-parameters for model training and generalization. However, current hand-designed parametric learning rate schedules offer limited flexibility and the predefined schedule may not match the training dynamics of high dimensional and non-convex optimization problems. In …

FedLion improves Federated Learning by speeding up convergence and reducing communication costs.

We study two time-scale linear stochastic approximation algorithms, which can be used to model well-known reinforcement learning algorithms such as GTD, GTD2, and TDC. We present finite-time performance bounds for the case where the learning rate is fixed. The key idea in obtaining these bounds is to use a Lyapunov fun…

This work addresses various open questions in the theory of active learning for nonparametric classification. Our contributions are both statistical and algorithmic: -We establish new minimax-rates for active learning under common \textit{noise conditions}. These rates display interesting transitions -- due to the inte…

We propose a statistical adaptive procedure called SALSA for automatically scheduling the learning rate (step size) in stochastic gradient methods. SALSA first uses a smoothed stochastic line-search procedure to gradually increase the learning rate, then automatically switches to a statistical method to decrease the le…

Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Rec…

Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, espec…

From a simplified analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned. We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search. I…

New algorithm for adaptive experimental design in scientific settings.

New active learning algorithm adapts to data without strict assumptions.

New learning rates for embeddings in RKHSs, even when the target is not Hilbert-Schmidt.

In this paper, we consider multi-agent learning via online gradient descent in a class of games called $λ$-cocoercive games, a fairly broad class of games that admits many Nash equilibria and that properly includes unconstrained strongly monotone games. We characterize the finite-time last-iterate convergence rate for …

We present a novel method for convex unconstrained optimization that, without any modifications, ensures: (i) accelerated convergence rate for smooth objectives, (ii) standard convergence rate in the general (non-smooth) setting, and (iii) standard convergence rate in the stochastic optimization setting. To the best of…

Most methods for decision-theoretic online learning are based on the Hedge algorithm, which takes a parameter called the learning rate. In most previous analyses the learning rate was carefully tuned to obtain optimal worst-case performance, leading to suboptimal performance on easy instances, for example when there ex…

AdaS adapts SGD learning rate based on knowledge gain metrics.