arXiv research

New adaptive step-size method for convex optimization without tuning.

Stochastic Gradient Descent (SGD) is a popular tool in training large-scale machine learning models. Its performance, however, is highly variable, depending crucially on the choice of the step sizes. Accordingly, a variety of strategies for tuning the step sizes have been proposed, ranging from coordinate-wise approach…

The variance reduction class of algorithms including the representative ones, SVRG and SARAH, have well documented merits for empirical risk minimization problems. However, they require grid search to tune parameters (step size and the number of iterations per inner loop) for optimal performance. This work introduces `…

ATLAS adapts HMC step size and trajectory length for complex geometries.

Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instabil…

One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a…

New method predicts and optimizes matrix recovery from noisy measurements.

We analyze SGAs for statistical inference via asymptotics, improving tuning methods.

Guarantees for tuning step size using meta-gradient descent.

Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.

The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by …

Adaptive step-size method improves compressed SGD performance in machine learning.

AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.

Proposes a neural network for learning step-size policies for L-BFGS optimization.

New algorithms for sampling and optimization without tuning.

New method tunes SMC samplers efficiently without high costs.

ProxSPS improves on SPS for regularization tasks, offering better stability and performance.

Proposes an automatic cyclical scheduling for gradient-based discrete sampling.

This paper explores the problem of learning transforms for image compression via autoencoders. Usually, the rate-distortion performances of image compression are tuned by varying the quantization step size. In the case of autoen-coders, this in principle would require learning one transform per rate-distortion point at…

Stochastic (sub)gradient methods require step size schedule tuning to perform well in practice. Classical tuning strategies decay the step size polynomially and lead to optimal sublinear rates on (strongly) convex problems. An alternative schedule, popular in nonconvex optimization, is called \emph{geometric step decay…

New algorithm improves stability of optimization algorithms by adapting step-size.

New adaptive scheduler improves SAM for better model training.

We propose a stochastic optimization method for minimizing loss functions, expressed as an expected value, that adaptively controls the batch size used in the computation of gradient approximations and the step size used to move along such directions, eliminating the need for the user to tune the learning rate. The pro…

New algorithm tunes SGMCMC hyperparameters for scalable Bayesian inference.

A new method approximates expected empirical loss for stochastic deep learning tasks.

SGD converges almost surely in non-convex problems, avoiding saddle points and accelerating convergence.

ICE-Pruning accelerates deep neural network pruning by 9.61x.

We consider $d$-dimensional linear stochastic approximation algorithms (LSAs) with a constant step-size and the so called Polyak-Ruppert (PR) averaging of iterates. LSAs are widely applied in machine learning and reinforcement learning (RL), where the aim is to compute an appropriate $θ_{*} \in \mathbb{R}^d$ (that is a…

New TD method stabilizes average-reward learning.

New step-size methods improve SHB convergence for stochastic optimization.

StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective for general nonconvex optimization, has received great attention due to its simple recursive framework for updating stochastic gradient estimates. The performance of SARAH significantly depends o…

Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs). Even for the baseline of a constant learning rate, it is non-trivial to choose a good constant value for training a DNN. Dynamic learning rates involve multi-step tuning of LR values at various stages of th…

In reinforcement learning, the TD($λ$) algorithm is a fundamental policy evaluation method with an efficient online implementation that is suitable for large-scale problems. One practical drawback of TD($λ$) is its sensitivity to the choice of the step-size. It is an empirically well-known fact that a large step-size l…

High sensitivity of neural architecture search (NAS) methods against their input such as step-size (i.e., learning rate) and search space prevents practitioners from applying them out-of-the-box to their own problems, albeit its purpose is to automate a part of tuning process. Aiming at a fast, robust, and widely-appli…

Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.

Adaptive gradient methods converge faster with over-parameterization and line-search.

Paper introduces a privacy-preserving line search method for optimization.

Meta-strategy learns tuning parameters for online learning methods.

Delayed rejection HMC improves sampling efficiency for multiscale distributions.

Annealed Langevin dynamics improves sampling from composite scores in SBI.

Predictions and predictive knowledge have seen recent success in improving not only robot control but also other applications ranging from industrial process control to rehabilitation. A property that makes these predictive approaches well suited for robotics is that they can be learned online and incrementally through…

Applying standard Markov chain Monte Carlo (MCMC) algorithms to large data sets is computationally expensive. Both the calculation of the acceptance probability and the creation of informed proposals usually require an iteration through the whole data set. The recently proposed stochastic gradient Langevin dynamics (SG…

We present a novel Metropolis-Hastings method for large datasets that uses small expected-size minibatches of data. Previous work on reducing the cost of Metropolis-Hastings tests yield variable data consumed per sample, with only constant factor reductions versus using the full dataset for each sample. Here we present…

GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.

LoRA enhances model adaptability without increasing parameters.

We develop a projected Nesterov's proximal-gradient (PNPG) approach for sparse signal reconstruction that combines adaptive step size with Nesterov's momentum acceleration. The objective function that we wish to minimize is the sum of a convex differentiable data-fidelity (negative log-likelihood (NLL)) term and a conv…

Stochastic proximal point algorithm with momentum converges faster and is more stable than standard methods.

We propose a statistical adaptive procedure called SALSA for automatically scheduling the learning rate (step size) in stochastic gradient methods. SALSA first uses a smoothed stochastic line-search procedure to gradually increase the learning rate, then automatically switches to a statistical method to decrease the le…