arXiv research

New algorithm improves stability of optimization algorithms by adapting step-size.

Adaptive step-size improves optimization in complex geometries.

New adaptive step-size method for convex optimization without tuning.

Proposes a neural network for learning step-size policies for L-BFGS optimization.

New convergence analysis for ADAM algorithm in non-convex optimization with adaptive step size.

Step decay schedules improve convergence in non-convex optimization.

Develops a generalized version of Chung's Lemma for stochastic optimization methods.

A new scaling law predicts optimal batch size for training models.

A major challenge in current optimization research for deep learning is to automatically find optimal step sizes for each update step. The optimal step size is closely related to the shape of the loss in the update step direction. However, this shape has not yet been examined in detail. This work shows empirically that…

Study optimizes step size for Metropolis algorithm in non-identifiable cases.

A new method approximates expected empirical loss for stochastic deep learning tasks.

Random Function Descent improves optimization in high dimensions.

A recent algorithmic family for distributed optimization, DIGing's, have been shown to have geometric convergence over time-varying undirected/directed graphs. Nevertheless, an identical step-size for all agents is needed. In this paper, we study the convergence rates of the Adapt-Then-Combine (ATC) variation of the DI…

New SPS variant improves non-smooth optimization without small gradients.

New convergence results for NGVI with various step sizes and sample sizes.

Proposes an exponentially increasing step-size for faster parameter estimation in statistical models.

In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…

Adaptive step-size method improves compressed SGD performance in machine learning.

The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.

We analyze the learning properties of the stochastic gradient method when multiple passes over the data and mini-batches are allowed. We study how regularization properties are controlled by the step-size, the number of passes and the mini-batch size. In particular, we consider the square loss and show that for a unive…

We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-convex case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis…

New step-size methods improve SHB convergence for stochastic optimization.

Negative step sizes improve second-order methods for neural networks.

Paper introduces a privacy-preserving line search method for optimization.

Avare improves optimization and sampling with adaptive importance sampling.

Proposes a new adaptive learning rate for SGD.

Optimistic method adapted for faster convex-concave min-max problems.

New method predicts and optimizes matrix recovery from noisy measurements.

Sparse coding is typically solved by iterative optimization techniques, such as the Iterative Shrinkage-Thresholding Algorithm (ISTA). Unfolding and learning weights of ISTA using neural networks is a practical way to accelerate estimation. In this paper, we study the selection of adapted step sizes for ISTA. We show t…

We propose a new stochastic first-order algorithmic framework to solve stochastic composite nonconvex optimization problems that covers both finite-sum and expectation settings. Our algorithms rely on the SARAH estimator introduced in (Nguyen et al, 2017) and consist of two steps: a proximal gradient and an averaging s…

This work provides formal guarantees for heuristic optimization methods in machine learning.

In this paper, we propose a simple, fast and easy to implement algorithm LOSSGRAD (locally optimal step-size in gradient descent), which automatically modifies the step-size in gradient descent during neural networks training. Given a function $f$, a point $x$, and the gradient $\nabla_x f$ of $f$, we aim to find the s…

Guarantees for tuning step size using meta-gradient descent.

Recently it has been shown that the step sizes of a family of variance reduced gradient methods called the JacSketch methods depend on the expected smoothness constant. In particular, if this expected smoothness constant could be calculated a priori, then one could safely set much larger step sizes which would result i…

Quantized Stochastic Primal-Dual Methods for Distributed Optimization

One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a…

The variance reduction class of algorithms including the representative ones, SVRG and SARAH, have well documented merits for empirical risk minimization problems. However, they require grid search to tune parameters (step size and the number of iterations per inner loop) for optimal performance. This work introduces `…

Sparse Polyak improves high-dimensional statistical estimation.

This manuscript shows that AdaBoost and its immediate variants can produce approximate maximum margin classifiers simply by scaling step size choices with a fixed small constant. In this way, when the unscaled step size is an optimal choice, these results provide guarantees for Friedman's empirically successful "shrink…

StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective for general nonconvex optimization, has received great attention due to its simple recursive framework for updating stochastic gradient estimates. The performance of SARAH significantly depends o…

Self-attention optimizers converge to optimal weights, revealing bias patterns.

RSGDA improves convergence rates for nonconvex-strongly concave optimization.

Investment strategy depends on many factors for venture capital funds.

SGD converges almost surely in non-convex problems, avoiding saddle points and accelerating convergence.

Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.

In empirical risk optimization, it has been observed that stochastic gradient implementations that rely on random reshuffling of the data achieve better performance than implementations that rely on sampling the data uniformly. Recent works have pursued justifications for this behavior by examining the convergence rate…

Chebyshev steps improve convergence in deep-unfolded gradient descent.

A common way to train neural networks is the Backpropagation. This algorithm includes a gradient descent method, which needs an adaptive step size. In the area of neural networks, the ADAM-Optimizer is one of the most popular adaptive step size methods. It was invented in \cite{Kingma.2015} by Kingma and Ba. The $5865$…