arXiv research

Cyclical learning rates improve DRL performance without manual tuning.

Proposes a learning rate method for shallow nets based on gradient Lipschitz constant.

Two retraining techniques outperform fine-tuning in neural network pruning.

Learning rate annealing improves robustness in stochastic optimization.

Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs). Even for the baseline of a constant learning rate, it is non-trivial to choose a good constant value for training a DNN. Dynamic learning rates involve multi-step tuning of LR values at various stages of th…

Two methods for model adaptation compared; fine-tuning outperforms Best-of-N in realizable settings.

Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured pro…

Despite the development of numerous adaptive optimizers, tuning the learning rate of stochastic gradient methods remains a major roadblock to obtaining good practical performance in machine learning. Rather than changing the learning rate at each iteration, we propose an approach that automates the most common hand-tun…

We consider the problem of minimizing a convex risk with stochastic subgradients guaranteeing $ε$-locally differentially private ($ε$-LDP). While it has been shown that stochastic optimization is possible with $ε$-LDP via the standard SGD (Song et al., 2013), its convergence rate largely depends on the learning rate, w…

D-Adaptation automatically sets optimal learning rates without manual tuning.

New approach uses PDE learning for faster RL fine-tuning.

The performance of stochastic gradient descent (SGD) depends critically on how learning rates are tuned and decreased over time. We propose a method to automatically adjust multiple learning rates so as to minimize the expected error at any one time. The method relies on local gradient variations across samples. In our…

AutoSGD automatically adjusts learning rates for SGD.

Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several comm…

We propose a stochastic optimization method for minimizing loss functions, expressed as an expected value, that adaptively controls the batch size used in the computation of gradient approximations and the step size used to move along such directions, eliminating the need for the user to tune the learning rate. The pro…

Paper proves multiplicative weight updates can train neural networks without learning rate tuning.

Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD …

GOLS-I automatically determines learning rates for various neural network training algorithms.

New coin sampling method for Bayesian inference without learning rates.

Optimizing deep neural networks is largely thought to be an empirical process, requiring manual tuning of several hyper-parameters, such as learning rate, weight decay, and dropout rate. Arguably, the learning rate is the most important of these to tune, and this has gained more attention in recent works. In this paper…

New algorithms learn latent variable models without tuning, outperforming existing methods.

New algorithms optimize without tuning, matching tuned SGD performance.

AutoGD automatically adjusts learning rates for gradient descent.

We consider online learning with linear models, where the algorithm predicts on sequentially revealed instances (feature vectors), and is compared against the best linear function (comparator) in hindsight. Popular algorithms in this framework, such as Online Gradient Descent (OGD), have parameters (learning rates), wh…

GALA adapts learning rates online by aligning gradients, improving deep learning model performance.

A new dynamic learning-rate scheme for optimization.

Hyperparameter tuning is a bothersome step in the training of deep learning models. One of the most sensitive hyperparameters is the learning rate of the gradient descent. We present the 'All Learning Rates At Once' (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learni…

Deep learning methods achieve state-of-the-art performance in many application scenarios. Yet, these methods require a significant amount of hyperparameters tuning in order to achieve the best results. In particular, tuning the learning rates in the stochastic optimization process is still one of the main bottlenecks. …

Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.

New methods reduce private federated learning communication automatically.

Improves risk control in predictions using semi-supervised calibration.

Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for…

Local update methods' performance depends on learning rates, affecting convergence rates and alignment with true loss.

New adaptive scheduler improves SAM for better model training.

TempBalance boosts model performance with low data.

New optimizer SF-NorMuon matches tuned AdamW across various horizons.

Prodigy estimates learning rate without tuning, improving convergence.

WSqD extends learning rate schedules for large model training without fixed horizons.

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…

FIRE PBT improves neural network training by focusing on long-term performance.

New study reveals surprising adaptive rates in model selection for transfer learning.

Bayesian optimisation for dynamically adjusting learning rates in machine learning models.

Model-agnostic meta-learning (MAML) is a meta-learning technique to train a model on a multitude of learning tasks in a way that primes the model for few-shot learning of new tasks. The MAML algorithm performs well on few-shot learning problems in classification, regression, and fine-tuning of policy gradients in reinf…

Meta-strategy learns tuning parameters for online learning methods.

PANDA improves linear discriminant analysis in high dimensions with minimal tuning.

New insights into optimizing Local SGD's outer optimizer for faster convergence.

New algorithm closes empirical gap in PFSGD performance.

A method learns common bias for multiple low-variance tasks without hyper-parameter tuning.