Smoothed SGD improves quantile estimation without crossing curves.
problem Estimating quantiles without crossing estimated curves.
method Smoothed SGD algorithm with Bahadur representation and Gaussian approximation.
result Smoothed SGD provides non-asymptotic tail probability bounds and a Gaussian approximation for quantile estimates.
Paper proposes an online estimator for covariance matrix of SGD iterates.
problem Quantifying variability and randomness of SGD-based estimates in online learning.
method Proposes a fully online estimator for covariance matrix of ASGD using SGD iterates.
result Establishes consistency of the online estimator and shows comparable convergence rate to offline methods.
Two new covariance estimators for ROOT-SGD improve statistical inference.
problem Uncertainty measurement for ROOT-SGD's normal distribution estimator.
method Developed two covariance estimators: plug-in and Hessian-free.
result Hessian-free estimator is asymptotically consistent and Hessian-free.
Paper develops methods for statistical inference with SGD in nonconvex optimization.
problem Statistical inference for nonconvex optimization problems.
method Proposes two online inferential procedures combining SGD and bootstrap techniques.
result Establishes error convergence rates and asymptotically valid bootstrap confidence intervals.
Doubly SGD improves convergence for intractable objective optimization.
problem Optimizing objectives in sum of intractable expectations.
method Doubly SGD with doubly stochastic gradients and independent minibatching.
result Established convergence of doubly SGD under general conditions, including dependent component gradient estimators.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
This paper analyzes Local SGD for federated learning, achieving both statistical and communication efficiency.
problem Statistical estimation and inference in federated learning with decentralized data.
method Local SGD, a multi-round estimation procedure using intermittent communication.
result Local SGD achieves both statistical efficiency and communication efficiency.
SGD performs optimally on dependent data, achieving both estimation and regret.
problem Performance of SGD on temporally dependent data.
method Stochastic gradient descent with martingale-type and sequential dependence.
result SGD achieves optimal estimation and regret under various dependence structures.
This paper provides a framework to analyze stochastic gradient algorithms in a mean squared error (MSE) sense using the asymptotic normality result of the stochastic gradient descent (SGD) iterates. We perform this analysis by taking the asymptotic normality result and applying it to the finite iteration case. Specific…
New analysis of SGD with MCMC gradient estimator shows convergence rate and saddle point escape.
problem Analyzing SGD with MCMC gradient estimator under complex conditions.
method Introduced MCMC-SGD, analyzed convergence rate and saddle point escape using Bernstein inequality.
result Proven first order convergence rate O ( log K / n K ) O(\log K/\sqrt{n K}) O ( log K / n K ) and saddle point escape at least O ( ε − 11 / 2 log 2 ( 1 / ε ) ) O(ε^{-11/2}\log^{2}(1/ε) ) O ( ε − 11/2 log 2 ( 1/ ε )) steps. Study improves covariance estimation for SGD under Markovian data, matching best rates.
problem Improving covariance estimation for SGD in Markovian data settings.
method Online overlapping batch-means covariance estimator for SGD under Markovian sampling.
result Established convergence rates for covariance estimation under Markovian sampling.
Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these tw…
FSGD uses latent factors to scale SGD for high-dimensional learning.
problem Scalable optimization in high-dimensional machine learning.
method Factor-Augmented SGD (FSGD) that operates on streaming data.
result Established theoretical framework for latent factor estimation error in SGD.
Stochastic Gradient Descent (SGD) is one of the most widely used techniques for online optimization in machine learning. In this work, we accelerate SGD by adaptively learning how to sample the most useful training examples at each time step. First, we show that SGD can be used to learn the best possible sampling distr…
The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistica…
SGD benefits from a directional bias in kernel regression models.
problem Improving generalization in kernel regression models.
method Generalized directional bias property of SGD in kernel regression.
result SGD converges along the eigenvector of the largest eigenvalue of the Gram matrix.
Improved Clipped-SGD achieves near-optimal heavy-tailed statistical estimation in streaming settings.
problem High-dimensional heavy-tailed statistical estimation in streaming with memory constraints.
method Stochastic convex optimization with Clipped-SGD, proving near-optimal sub-Gaussian statistical rates.
result Clipped-SGD achieves an error of T r ( Σ ) + T r ( Σ ) ∥ Σ ∥ 2 log ( log ( T ) δ ) T \sqrt{\frac{\mathsf{Tr}(Σ)+\sqrt{\mathsf{Tr}(Σ)\|Σ\|_2}\log(\frac{\log(T)}δ)}{T}} T Tr ( Σ ) + Tr ( Σ ) ∥Σ ∥ 2 l o g ( δ l o g ( T ) ) with probability 1 − δ 1-δ 1 − δ . A method for efficient statistical inference from online algorithms.
problem Computational constraints in online algorithms make traditional variance estimation difficult.
method HulC method that wraps around online algorithms to produce valid confidence regions.
result The HulC method produces asymptotically valid confidence regions for online algorithms.
SGD-trained deep nets have bounds on their generalization error.
problem Bounding generalization error for deep neural networks trained by SGD.
method Combining dynamical control of parameter norms and Rademacher complexity estimates.
result Explicit bounds depend on loss trajectory, work for various architectures.
New CLT for SGD in high-dimensional regression provides online inference.
problem Quantifying uncertainty in SGD for high-dimensional regression.
method Established a high-dimensional CLT for online SGD iterates.
result Developed an online approach for estimating variance in CLT.
Paper proves CLT for quantile SGD with constant learning rate.
problem Quantile estimation via SGD with non-smooth, non-strongly convex loss.
method Viewed as a Markov chain, derived stationary distribution, analyzed MGF, proved CLT.
result Centered and standardized stationary distribution converges to Gaussian as η i g h t a r r o w 0 η
ightarrow0 η i g h t a r r o w 0 . In many applications involving large dataset or online updating, stochastic gradient descent (SGD) provides a scalable way to compute parameter estimates and has gained increasing popularity due to its numerical convenience and memory efficiency. While the asymptotic properties of SGD-based estimators have been establi…
New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.
problem The generalization benefits of decentralized learning.
method Proved D-SGD implicitly minimizes SAM's loss function.
result Decentralized SGD and Average-direction SAM are asymptotically equivalent.
SGD improves generalization by using gradient variability as a proxy for data randomness.
problem Improving generalization in machine learning models trained with stochastic gradient descent.
method Bootstrap perspective on SGD, analyzing gradient variability and algorithmic variability.
result SGD avoids spurious solutions and improves generalization by implicitly regularizing the trace of the gradient covariance matrix.
A new method for statistical inference using SGD under φ φ φ -mixing data.
problem Valid statistical inference for time series data with general correlation.
method Proposes a mini-batch SGD estimator and associated mini-batch bootstrap procedure for φ φ φ -mixing data. result The proposed method constructs valid confidence intervals for φ φ φ -mixing data. New sparsification technique for SGD reduces communication costs.
problem High communication costs in distributed SGD for large-scale models.
method Statistical estimation model for sparsity and skewness of stochastic gradients.
result Concatenated top-k and random-k sparsification outperforms individual methods.
Nonlinear SGD achieves high-probability rates in non-convex optimization with heavy-tailed noise.
problem Optimization in non-convex problems with heavy-tailed noise.
method General nonlinear framework for SGD, including symmetrization techniques.
result Achieves O ~ ( t − 1 / 2 ) \widetilde{\mathcal{O}}(t^{-1/2}) O ( t − 1/2 ) rate for heavy-tailed noise. AdaSGD combines SGD and Adam benefits, eliminating the need for transition.
problem Understanding when to transition from Adam to SGD for optimal performance.
method Adapting a single global learning rate for SGD (AdaSGD).
result AdaSGD combines the benefits of both SGD and Adam, improving convergence and generalization.
Error estimates found between SGD with momentum and Langevin diffusion.
problem Quantifying the difference between SGD with momentum and Langevin diffusion.
method Established error estimates using 1-Wasserstein and total variation distances.
result Quantitative error estimates between SGD with momentum and underdamped Langevin diffusion.
New bounds for SGD in high dimensions improve inference efficiency.
problem Quantifying uncertainty in high-dimensional SGD.
method Established non-asymptotic Berry--Esseen bounds for online least-squares SGD.
result Gaussian Central Limit Theorem holds for t ≳ d 1 + δ t \gtrsim d^{1+δ} t ≳ d 1 + δ , extending dimensional scaling. Paper develops methods for statistical inference in SGD with infinite variance.
problem Challenges in statistical inference for SGD with infinite variance.
method Model-agnostic methodology based on weak convergence and subsampling calibration.
result Asymptotically valid confidence regions for SGD in both finite and infinite variance regimes.
New algorithm optimizes linear system estimation from single trajectory.
problem Estimating LTI systems from a single trajectory.
method SGD with Reverse Experience Replay ( S G D − R E R \mathsf{SGD}-\mathsf{RER} SGD − RER ) result Optimal guarantees for parameter and prediction errors.
The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.
problem Stochastic optimization in a streaming setting with time-dependent and biased gradient estimates.
method Analysis of several first-order methods including SGD, mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages.
result Time-varying mini-batch SGD methods can break long- and short-range dependence structures, and biased SGD methods can achieve comparable performance to their unbiased counterparts.
New SGD algorithm finds critical points faster with second-order corrections.
problem Finding critical points in non-convex optimization efficiently.
method Uses Hessian-vector products to correct momentum bias in SGD.
result Finds ε ε ε -critical points in O ( ε − 3 ) O(ε^{-3}) O ( ε − 3 ) time. Stochastic gradient descent (SGD) is the workhorse of modern machine learning. Sometimes, there are many different potential gradient estimators that can be used. When so, choosing the one with the best tradeoff between cost and variance is important. This paper analyzes the convergence rates of SGD as a function of ti…
The paper analyzes two ISGD modes for statistical inference, deriving error bounds and confidence intervals.
problem Statistical inference with implicit SGD for smooth convex functions.
method Proximal Robbins-Monro (proxRM) and proximal Polyak-Ruppert (proxPR) procedures for ISGD.
result Derives non-asymptotic error bounds and confidence interval estimators for model parameters.
Artificial intelligence is revolutionizing our lives at an ever increasing pace. At the heart of this revolution is the recent advancements in deep neural networks (DNN), learning to perform sophisticated, high-level tasks. However, training DNNs requires massive amounts of data and is very computationally intensive. G…
SGD transitions between maxima and minima with varying time scales.
problem Understanding SGD's behavior near critical points in noisy landscapes.
method Analyzing SGD convergence and escape dynamics in 1D landscapes with infinite- and finite-variance noise.
result SGD reliably moves to the basin's minimum unless close to a local maximum, where it can linger.
Adapts SGD to noise and problem specifics for faster convergence.
problem Minimizing smooth, strongly-convex functions with varying noise and problem constants.
method Adaptive SGD with exponentially decreasing step-sizes, Nesterov acceleration, and stochastic line-search.
result Achieves near-optimal convergence rates without knowing noise or problem specifics.
This work improves SGD minibatch sampling using determinantal point processes based on orthogonal polynomials.
problem Improving variance reduction in stochastic gradient descent (SGD) for large datasets.
method Orthogonal polynomial-based determinantal point processes for sampling minibatches in SGD.
result DPP minibatches lead to a smaller mean square approximation error than uniform minibatches.
SGD converges to an invariant distribution with sub-Gaussian or sub-exponential properties.
problem Optimizing smooth and strongly convex objectives using SGD.
method Analysis through Markov chains, focusing on convergence and concentration properties.
result SGD iterates and their invariant limit distribution inherit sub-Gaussian or sub-exponential concentration properties.
Machine learning, especially deep neural networks, has been rapidly developed in fields including computer vision, speech recognition and reinforcement learning. Although Mini-batch SGD is one of the most popular stochastic optimization methods in training deep networks, it shows a slow convergence rate due to the larg…
Proposes a method to reduce parallel complexity of MLMC in SGD.
problem Poor scalability of MLMC in SGD on parallel platforms.
method Proposes a delayed MLMC gradient estimator to reduce parallel complexity.
result Proves reduction in average parallel complexity per iteration at the cost of slightly worse convergence rate.
We present a novel method for frequentist statistical inference in M M M -estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlen…
New method for uncertainty quantification in private SGD.
problem Uncertainty quantification for differentially private SGD.
method Block bootstrap for SGD under local differential privacy.
result Valid method for uncertainty quantification in private SGD.
This work analyzes DP-SGD for online LDP problems with practical convergence rates.
problem Analyzing DP-SGD for online LDP problems with practical convergence rates.
method Developed a general framework for online LDP model in stochastic optimization problems, conducted non-asymptotic convergence analysis.
result Comprehensive non-asymptotic convergence analysis of the proposed estimators in finite-sample situations.
Vanilla SGD learns SIM from anisotropic data without explicit covariance estimation.
problem Learning SIM from anisotropic Gaussian inputs.
method Vanilla Stochastic Gradient Descent (SGD) trained on SIM with anisotropic input.
result Vanilla SGD adapts to anisotropic data's covariance structure.
Local SGD proves efficient in overparameterized linear regression.
problem Efficiently learning overparameterized linear models in distributed settings.
method Distributed SGD (DSGD) with overparameterized models.
result Excess risk of SGD is smaller than ridge regression in the same sample complexity.