arXiv research

The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.

Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning

Paper develops methods for statistical inference in SGD with infinite variance.

Smoothed SGD improves quantile estimation without crossing curves.

We provide non-asymptotic convergence rates of the Polyak-Ruppert averaged stochastic gradient descent (SGD) to a normal random vector for a class of twice-differentiable test functions. A crucial intermediate step is proving a non-asymptotic martingale central limit theorem (CLT), i.e., establishing the rates of conve…

New convergence rates for SGD under heavy-tailed noise with infinite variance.

Study optimizes decisions in real-time using inexact simulation solutions.

The paper explores how dynamic preconditioning affects the CLT in online averaging.

Study on stochastic approximation with Polyak-Ruppert averaging for linear systems.

SGD converges to an invariant distribution with sub-Gaussian or sub-exponential properties.

New method for online inference using SGD with random scaling.

A method for efficient statistical inference from online algorithms.

The paper analyzes two ISGD modes for statistical inference, deriving error bounds and confidence intervals.

Study Q-learning with averaging for reinforcement learning, proving efficient inference and error bounds.

Paper proves CLTs for Q-learning with asynchronous updates.

TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates

New streaming methods improve convergence rates for optimization problems.

The paper analyzes SGD with Richardson-Romberg extrapolation for convex optimization problems.

Paper improves confidence intervals for LSA with multiplier bootstrap.

Paper improves confidence set construction for SGD using multiplier bootstrap.

Paper analyzes normal approximation for two-timescale stochastic algorithms, revealing interaction between fast and slow timescales.

We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this …

Stochastic gradient descent on l1 loss converges to true parameter in online robust regression.

The paper offers precise bounds for averaged LSA iterates in linear systems.

Averaged SGD optimizes a smoothed objective, leading to better generalization.

New method for zeroth-order stochastic gradient algorithms provides confidence intervals.

Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.

Paper analyzes how EMA improves SGD in linear regression.

Paper develops bounds for stochastic approximation with averaging.

ADPSGD optimizes SGD convergence by varying averaging periods based on model variance.

Partial model averaging improves Federated Learning performance.

This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…

Simplified analysis of SGD for linear regression with weight averaging.

Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…

SGD in linear regression overfits but performs well due to bias-variance trade-off.

In this paper, we study a simple algorithm to construct asymptotically valid confidence regions for model parameters using the batch means method. The main idea is to cancel out the covariance matrix which is hard/costly to estimate. In the process of developing the algorithm, we establish process-level functional cent…

A2SGD reduces distributed SGD communication to O(1) per worker.

While stochastic gradient descent (SGD) is one of the major workhorses in machine learning, the learning properties of many practically used variants are poorly understood. In this paper, we consider least squares learning in a nonparametric setting and contribute to filling this gap by focusing on the effect and inter…

MDA optimizer performs similarly to SGD+M in CV and Adam in NLP.

Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth object…

New insights into using momentum for non-convex optimization.

Unified analysis of finite weight averaging methods in deep learning.

This paper analyzes SGD with increasingly weighted averaging for optimization and generalization.

New bounds for SGD show improved performance in various settings.

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conv…

Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these tw…

New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.

New method averages SGD iterates to achieve adjustable regularization.