arXiv research

Last SGD iterate bounds for overparameterized linear regression.

This paper provides a framework to analyze stochastic gradient algorithms in a mean squared error (MSE) sense using the asymptotic normality result of the stochastic gradient descent (SGD) iterates. We perform this analysis by taking the asymptotic normality result and applying it to the finite iteration case. Specific…

Paper proposes an online estimator for covariance matrix of SGD iterates.

New study shows faster convergence of SGD and Kaczmarz methods.

To accelerate the training of machine learning models, distributed stochastic gradient descent (SGD) and its variants have been widely adopted, which apply multiple workers in parallel to speed up training. Among them, Local SGD has gained much attention due to its lower communication cost. Nevertheless, when the data …

New bounds on SGD's final iterate convergence rate in constant dimension.

New bounds for SGD show improved performance in various settings.

SGD's performance improves with critical batch size, minimizing SFO complexity.

Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these tw…

SGD in linear regression overfits but performs well due to bias-variance trade-off.

SGD generalizes better than GD with similar number of iterations.

Mixed-precision CA-SGD for generalized linear models on GPUs

Minimax optimal convergence rates for classes of stochastic convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, SGD's final iterate behavior has received much less attention desp…

We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent~(SGD) methods. To reduce the computation costs pertaining to the projections, we pro…

SGD works well with large learning rates at the edge of stability.

The paper analyzes SGD with dropout regularization in linear models, proving asymptotic properties and providing inference tools.

In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…

Study on adversarial training dynamics in high dimensions using SGD.

New method averages SGD iterates to achieve adjustable regularization.

SGD achieves near optimal convergence rate in smooth interpolation regime.

This work addresses the convergence of SGD's final iterate without restrictive assumptions.

New privacy bounds for DP-SGD's last iterate, even with cyclic sampling.

In this note we give a simple proof for the convergence of stochastic gradient (SGD) methods on $μ$-convex functions under a (milder than standard) $L$-smoothness assumption. We show that for carefully chosen stepsizes SGD converges after $T$ iterations as $O\left( LR^2 \exp \bigl[-\fracμ{4L}T\bigr] + \frac{σ^2}{μT} \r…

SGD generalization bounds derived from information theory.

New bounds for SGD in high dimensions improve inference efficiency.

Stochastic Gradient Descent (SGD) has become one of the most popular optimization methods for training machine learning models on massive datasets. However, SGD suffers from two main drawbacks: (i) The noisy gradient updates have high variance, which slows down convergence as the iterates approach the optimum, and (ii)…

Paper proposes a coupling-based diagnostic for SGD stepsize optimization.

With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are l…

This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…

This paper analyzes multi-pass SGD for least squares, improving generalization bounds.

AutoSGD automatically adjusts learning rates for SGD.

SGD performs optimally on dependent data, achieving both estimation and regret.

OptEx accelerates first-order optimization with parallelized iterations.

This paper analyzes Local SGD for federated learning, achieving both statistical and communication efficiency.

New CLT for SGD in high-dimensional regression provides online inference.

SGD converges to optimal solution in perfect data fitting problem.

ROOT-SGD solves convex optimization problems with optimal nonasymptotic and near-optimal asymptotic performance.

The paper analyzes convergence rates for SGD and SHB methods.

SGD with mini-batches can solve convex low-rank matrix problems efficiently.

A new algorithm combines SGD and Thompson Sampling for contextual bandits.

Adaptive SGD learns optimal batch size for strong convex functions.

We present an optimizer which uses Bayesian optimization to tune the system parameters of distributed stochastic gradient descent (SGD). Given a specific context, our goal is to quickly find efficient configurations which appropriately balance the load between the available machines to minimize the average SGD iteratio…

Derives scaling limits and fluctuations for SGD in high dimensions.

Study on SGD for overparameterized neural networks, focusing on convergence rates.

SGD converges to an invariant distribution with sub-Gaussian or sub-exponential properties.

We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…

Improved generalization bounds for SGD in non-convex learning.

New method shows how order of gradient updates impacts stability and convergence in deep learning.