arXiv research

Last SGD iterate bounds for overparameterized linear regression.

Minimax optimal convergence rates for classes of stochastic convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, SGD's final iterate behavior has received much less attention desp…

Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…

Learning rate annealing improves robustness in stochastic optimization.

Study on bias and extrapolation in LSA with Markovian data, showing bias reduction with Richardson-Romberg extrapolation.

Novel Adam-family method with decoupled weight decay for training neural networks.

Cyclic and randomized stepsizes can lead to heavier tails in SGD, improving generalization.

SGD with constant stepsize converges to a non-Gaussian limit near flat minima.

Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.

Paper examines constant stepsize in LSA for Markovian data inference.

Large stepsizes can accelerate gradient descent for logistic regression.

Large stepsize GD for logistic regression converges faster than expected.

We revisit the classical Douglas-Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the stepsizes, we aim at developing an adaptive stepsize rule. To that end, we take a closer look at a linear case of the problem a…

Paper proposes a coupling-based diagnostic for SGD stepsize optimization.

TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in o…

Variable metric proximal gradient (VM-PG) is a widely used class of convex optimization method. Lately, there has been a lot of research on the theoretical guarantees of VM-PG with different metric selections. However, most such metric selections are dependent on (an expensive) Hessian, or limited to scalar stepsizes l…

New algorithms eliminate stepsize tuning for bilevel optimization problems.

Stochastic (sub)gradient methods require step size schedule tuning to perform well in practice. Classical tuning strategies decay the step size polynomially and lead to optimal sublinear rates on (strongly) convex problems. An alternative schedule, popular in nonconvex optimization, is called \emph{geometric step decay…

New adaptive stepsize method for stochastic approximation converges to target point.

Study on nonsmooth contractive SA with constant stepsize and Q-learning.

Adaptive-stepsize MCMC sampling inspired by Adam optimizer.

Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.

Adaptive stepsizing improves sampling in Bayesian neural networks.

This work analyzes $Q$-learning with adaptive stepsizes for finite-time convergence.

Study Whittle index learning algorithms for restless bandits with constant stepsizes.

Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several years, a plethora of adaptive gradient-based algorithms have emerged to ameliorate t…

In this paper we investigate the life-span of classical solutions to the hyperbolic geometric flow in two space variables with slow decay initial data. By establishing some new estimates on the solutions of linear wave equations in two space variables, we give a lower bound of the life-span of classical solutions to th…

Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.

Large GD stepsizes improve margins and speed up training for non-homogeneous networks.

The paper improves convergence for linear systems using entropic mirror descent with Polyak stepsizes.

OSGM uses online learning to adapt stepsize for faster convergence.

New analysis improves SGD for robust and quantile regression with sub-quadratic convergence.

Optimal multistage method solves noisy minimax problems.

Locally adaptive federated learning improves convergence in distributed machine learning.

Gradient-based temporal difference (GTD) algorithms are widely used in off-policy learning scenarios. Among them, the two time-scale TD with gradient correction (TDC) algorithm has been shown to have superior performance. In contrast to previous studies that characterized the non-asymptotic convergence rate of TDC only…

This work accelerates gradient descent with anytime convergence guarantees.

Approximate dynamic programming (ADP) has proven itself in a wide range of applications spanning large-scale transportation problems, health care, revenue management, and energy systems. The design of effective ADP algorithms has many dimensions, but one crucial factor is the stepsize rule used to update a value functi…

Gradient descent converges with arbitrary stepsize for separable data under Fenchel-Young losses.

Proposes a new adaptive gradient method based on gradient differences.

The paper provides bounds for LSA with fixed stepsizes under random estimates.

Paper tackles dynamic pricing in a geometrically decaying environment, achieving better occupancy with lower rates.

Enhanced ROOT-SGD optimizes stochastic optimization with diminishing stepsizes.

GD with large, adaptive stepsizes achieves optimal risk in logistic regression.

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced op…

New method finds precise late-time behavior of wave equations.

In this article, we derive a new stepsize adaptation for the normalized least mean square algorithm (NLMS) by describing the task of linear acoustic echo cancellation from a Bayesian network perspective. Similar to the well-known Kalman filter equations, we model the acoustic wave propagation from the loudspeaker to th…

We consider any pseudo holomorphic integral 2-cycle in an arbitrary almost complex manifold and perform a blow up analysis at an arbitrary point. Building upon a pseudo algebraic blow up (previously introduced by the author) we prove a geometric rate of decay for the mass ratio towards the limiting density, with an exp…