arXiv research

Gradient descent converges with arbitrary stepsize for separable data under Fenchel-Young losses.

The paper improves convergence for linear systems using entropic mirror descent with Polyak stepsizes.

Cyclic and randomized stepsizes can lead to heavier tails in SGD, improving generalization.

We propose a general yet simple theorem describing the convergence of SGD under the arbitrary sampling paradigm. Our theorem describes the convergence of an infinite array of variants of SGD, each of which is associated with a specific probability law governing the data selection rule used to form mini-batches. This is…

Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.

Paper examines constant stepsize in LSA for Markovian data inference.

Last SGD iterate bounds for overparameterized linear regression.

Large stepsizes can accelerate gradient descent for logistic regression.

Large stepsize GD for logistic regression converges faster than expected.

We revisit the classical Douglas-Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the stepsizes, we aim at developing an adaptive stepsize rule. To that end, we take a closer look at a linear case of the problem a…

Paper proposes a coupling-based diagnostic for SGD stepsize optimization.

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in o…

Variable metric proximal gradient (VM-PG) is a widely used class of convex optimization method. Lately, there has been a lot of research on the theoretical guarantees of VM-PG with different metric selections. However, most such metric selections are dependent on (an expensive) Hessian, or limited to scalar stepsizes l…

New algorithms eliminate stepsize tuning for bilevel optimization problems.

New adaptive stepsize method for stochastic approximation converges to target point.

Study on nonsmooth contractive SA with constant stepsize and Q-learning.

Adaptive-stepsize MCMC sampling inspired by Adam optimizer.

Gradient descent recovers principal components of overparametrized asymmetric matrices without explicit regularization.

Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.

Adaptive stepsizing improves sampling in Bayesian neural networks.

This work analyzes $Q$-learning with adaptive stepsizes for finite-time convergence.

Study Whittle index learning algorithms for restless bandits with constant stepsizes.

Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several years, a plethora of adaptive gradient-based algorithms have emerged to ameliorate t…

Study on bias and extrapolation in LSA with Markovian data, showing bias reduction with Richardson-Romberg extrapolation.

Large GD stepsizes improve margins and speed up training for non-homogeneous networks.

OSGM uses online learning to adapt stepsize for faster convergence.

Gradient-based temporal difference (GTD) algorithms are widely used in off-policy learning scenarios. Among them, the two time-scale TD with gradient correction (TDC) algorithm has been shown to have superior performance. In contrast to previous studies that characterized the non-asymptotic convergence rate of TDC only…

This work accelerates gradient descent with anytime convergence guarantees.

Approximate dynamic programming (ADP) has proven itself in a wide range of applications spanning large-scale transportation problems, health care, revenue management, and energy systems. The design of effective ADP algorithms has many dimensions, but one crucial factor is the stepsize rule used to update a value functi…

Proposes a new adaptive gradient method based on gradient differences.

The paper provides bounds for LSA with fixed stepsizes under random estimates.

Enhanced ROOT-SGD optimizes stochastic optimization with diminishing stepsizes.

Gradient method converges locally linearly for overparameterized Gaussian mixtures.

GD with large, adaptive stepsizes achieves optimal risk in logistic regression.

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced op…

In this article, we derive a new stepsize adaptation for the normalized least mean square algorithm (NLMS) by describing the task of linear acoustic echo cancellation from a Bayesian network perspective. Similar to the well-known Kalman filter equations, we model the acoustic wave propagation from the loudspeaker to th…

Improves posterior approximation speed for Dirichlet process mixture models.

GD at EoS edge minimizes logistic loss without monotonic convergence.

New framework analyzes SGD dynamics in large samples and dimensions.

We propose methods for distributed graph-based multi-task learning that are based on weighted averaging of messages from other machines. Uniform averaging or diminishing stepsize in these methods would yield consensus (single task) learning. We show how simply skewing the averaging weights or controlling the stepsize a…

cKAM improves adaptive sampling by incorporating a cyclical stepsize scheme.

MARINA-P improves non-smooth federated optimization with adaptive stepsizes.

Unified framework for analyzing convergence of RSAs using Wasserstein divergence.

SGD with constant stepsize converges to a non-Gaussian limit near flat minima.

We present a strikingly simple proof that two rules are sufficient to automate gradient descent: 1) don't increase the stepsize too fast and 2) don't overstep the local curvature. No need for functional values, no line search, no information about the function except for the gradients. By following these rules, you get…

AdaSDBO solves decentralized bilevel optimization without problem parameters, achieving competitive performance.

Paper analyzes convergence of GDA for nonconvex-nonconcave minimax problems.

Recent years have witnessed the surge of asynchronous parallel (async-parallel) iterative algorithms due to problems involving very large-scale data and a large number of decision variables. Because of asynchrony, the iterates are computed with outdated information, and the age of the outdated information, which we cal…