arXiv research

Paper proposes a coupling-based diagnostic for SGD stepsize optimization.

Adaptive-stepsize MCMC sampling inspired by Adam optimizer.

Study on nonsmooth contractive SA with constant stepsize and Q-learning.

Adaptive stepsizing improves sampling in Bayesian neural networks.

New framework analyzes SGD dynamics in large samples and dimensions.

Approximate dynamic programming (ADP) has proven itself in a wide range of applications spanning large-scale transportation problems, health care, revenue management, and energy systems. The design of effective ADP algorithms has many dimensions, but one crucial factor is the stepsize rule used to update a value functi…

An algorithm is presented for momentum gradient descent optimization based on the first-order differential equation of the Newtonian dynamics. The fictitious mass is introduced to the dynamics of momentum for regularizing the adaptive stepsize of each individual parameter. The dynamic relaxation is adapted for stochast…

Enhanced ROOT-SGD optimizes stochastic optimization with diminishing stepsizes.

New algorithm accelerates optimization on Riemannian manifolds, including Wasserstein space.

Cyclic and randomized stepsizes can lead to heavier tails in SGD, improving generalization.

Gradient descent converges with arbitrary stepsize for separable data under Fenchel-Young losses.

The paper analyzes variance reduction in stochastic gradient Langevin dynamics.

AdaSDBO solves decentralized bilevel optimization without problem parameters, achieving competitive performance.

A new dynamic learning-rate scheme for optimization.

Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.

Paper examines constant stepsize in LSA for Markovian data inference.

Last SGD iterate bounds for overparameterized linear regression.

Large stepsizes can accelerate gradient descent for logistic regression.

Large stepsize GD for logistic regression converges faster than expected.

We revisit the classical Douglas-Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the stepsizes, we aim at developing an adaptive stepsize rule. To that end, we take a closer look at a linear case of the problem a…

New algorithm samples Bayesian neural networks for improved calibration.

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in o…

GD converges faster to flatter minima than gradient flow in shallow networks.

Variable metric proximal gradient (VM-PG) is a widely used class of convex optimization method. Lately, there has been a lot of research on the theoretical guarantees of VM-PG with different metric selections. However, most such metric selections are dependent on (an expensive) Hessian, or limited to scalar stepsizes l…

New algorithms improve sampling from complex distributions.

New algorithms eliminate stepsize tuning for bilevel optimization problems.

New adaptive stepsize method for stochastic approximation converges to target point.

This paper considers distributed online optimization with time-varying coupled inequality constraints. The global objective function is composed of local convex cost and regularization functions and the coupled constraint function is the sum of local convex functions. A distributed online primal-dual dynamic mirror des…

Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.

This work analyzes $Q$-learning with adaptive stepsizes for finite-time convergence.

Study Whittle index learning algorithms for restless bandits with constant stepsizes.

Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several years, a plethora of adaptive gradient-based algorithms have emerged to ameliorate t…

Study on bias and extrapolation in LSA with Markovian data, showing bias reduction with Richardson-Romberg extrapolation.

Large GD stepsizes improve margins and speed up training for non-homogeneous networks.

The paper improves convergence for linear systems using entropic mirror descent with Polyak stepsizes.

OSGM uses online learning to adapt stepsize for faster convergence.

Gradient-based temporal difference (GTD) algorithms are widely used in off-policy learning scenarios. Among them, the two time-scale TD with gradient correction (TDC) algorithm has been shown to have superior performance. In contrast to previous studies that characterized the non-asymptotic convergence rate of TDC only…

This work accelerates gradient descent with anytime convergence guarantees.

Proposes a new adaptive gradient method based on gradient differences.

The paper provides bounds for LSA with fixed stepsizes under random estimates.

GD with large, adaptive stepsizes achieves optimal risk in logistic regression.

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced op…

Study on adversarial training dynamics in high dimensions using SGD.

In this article, we derive a new stepsize adaptation for the normalized least mean square algorithm (NLMS) by describing the task of linear acoustic echo cancellation from a Bayesian network perspective. Similar to the well-known Kalman filter equations, we model the acoustic wave propagation from the loudspeaker to th…

Improves posterior approximation speed for Dirichlet process mixture models.

GD at EoS edge minimizes logistic loss without monotonic convergence.

DIGing-SGLD improves SGLD for scalable Bayesian learning in dynamic networks.

We propose methods for distributed graph-based multi-task learning that are based on weighted averaging of messages from other machines. Uniform averaging or diminishing stepsize in these methods would yield consensus (single task) learning. We show how simply skewing the averaging weights or controlling the stepsize a…