arXiv research

Adaptive stepsizing improves sampling in Bayesian neural networks.

We revisit the classical Douglas-Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the stepsizes, we aim at developing an adaptive stepsize rule. To that end, we take a closer look at a linear case of the problem a…

This work analyzes $Q$-learning with adaptive stepsizes for finite-time convergence.

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in o…

Proposes a new adaptive gradient method based on gradient differences.

New adaptive stepsize method for stochastic approximation converges to target point.

cKAM improves adaptive sampling by incorporating a cyclical stepsize scheme.

New algorithms eliminate stepsize tuning for bilevel optimization problems.

GD with large, adaptive stepsizes achieves optimal risk in logistic regression.

Adaptive-stepsize MCMC sampling inspired by Adam optimizer.

Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network par…

Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several years, a plethora of adaptive gradient-based algorithms have emerged to ameliorate t…

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced op…

Improved SGD with AdaGrad stepsizes adapts to unknown parameters and unbounded gradients.

OSGM uses online learning to adapt stepsize for faster convergence.

An algorithm is presented for momentum gradient descent optimization based on the first-order differential equation of the Newtonian dynamics. The fictitious mass is introduced to the dynamics of momentum for regularizing the adaptive stepsize of each individual parameter. The dynamic relaxation is adapted for stochast…

MARINA-P improves non-smooth federated optimization with adaptive stepsizes.

Variable metric proximal gradient (VM-PG) is a widely used class of convex optimization method. Lately, there has been a lot of research on the theoretical guarantees of VM-PG with different metric selections. However, most such metric selections are dependent on (an expensive) Hessian, or limited to scalar stepsizes l…

Improves posterior approximation speed for Dirichlet process mixture models.

In this article, we derive a new stepsize adaptation for the normalized least mean square algorithm (NLMS) by describing the task of linear acoustic echo cancellation from a Bayesian network perspective. Similar to the well-known Kalman filter equations, we model the acoustic wave propagation from the loudspeaker to th…

Improved SGD methods converge faster for nonconvex optimization.

Approximate dynamic programming (ADP) has proven itself in a wide range of applications spanning large-scale transportation problems, health care, revenue management, and energy systems. The design of effective ADP algorithms has many dimensions, but one crucial factor is the stepsize rule used to update a value functi…

AdaSDBO solves decentralized bilevel optimization without problem parameters, achieving competitive performance.

Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…

We consider a generic convex-concave saddle point problem with separable structure, a form that covers a wide-ranged machine learning applications. Under this problem structure, we follow the framework of primal-dual updates for saddle point problems, and incorporate stochastic block coordinate descent with adaptive st…

TiAda adapts adaptive gradient methods for nonconvex minimax optimization.

AdaGrad fails to adapt to Hölder-smoothness in composite optimization problems.

Cyclic and randomized stepsizes can lead to heavier tails in SGD, improving generalization.

New algorithm solves stochastic optimization problems with unknown gradients.

Reinforcement learning has attracted great attention recently, especially policy gradient algorithms, which have been demonstrated on challenging decision making and control tasks. In this paper, we propose an active multi-step TD algorithm with adaptive stepsizes to learn actor and critic. Specifically, our model cons…

New analysis reveals batch size effects on stochastic conditional gradient methods.

New algorithms improve SGD convergence and reduce variance for over-parameterized models.

Locally adaptive federated learning improves convergence in distributed machine learning.

Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.

Paper examines constant stepsize in LSA for Markovian data inference.

Last SGD iterate bounds for overparameterized linear regression.

NeAda solves nonconvex minimax optimization by balancing primal and dual variables adaptively.

Large stepsizes can accelerate gradient descent for logistic regression.

Large stepsize GD for logistic regression converges faster than expected.

Paper proposes a coupling-based diagnostic for SGD stepsize optimization.

Adaptive methods improve gradient descent and proximal gradient for convex optimization.

AdaBelief optimizes deep learning models with faster convergence and better stability.

A new method Expectigrad improves on Adam and RMSProp by reducing divergence and improving performance.

Study on nonsmooth contractive SA with constant stepsize and Q-learning.

Improved privacy and utility in machine learning with adaptive differential privacy.

New adaptive first-order methods improve on quasi-Newton variants.

Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.

NSGD-M optimizes machine learning models without hyperparameter tuning, even under relaxed smoothness.