Averaged SGD optimizes a smoothed objective, leading to better generalization.
problem Improving generalization performance in machine learning models.
method Analyzed the smoothed objective function of SGD and proved that averaged SGD can optimize this smoothed function efficiently.
result Averaged SGD can efficiently optimize a smoothed objective, leading to better generalization.
Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.
problem Improving convergence of SGD in various settings.
method Develops a general weighted averaging scheme for SGD and establishes asymptotic normality.
result Establishes asymptotic normality and optimality of weighted averaged SGD solutions.
Paper analyzes how EMA improves SGD in linear regression.
problem Understanding the effectiveness of EMA in training deep learning models.
method Established risk bound for online SGD with EMA in linear regression.
result SGD with EMA has smaller variance error and exponentially decaying bias error.
SWALP averages SGD iterates for low-precision training, improving scalability and performance.
problem Improving scalability and performance in low-precision training.
method Averages low-precision SGD iterates with a modified learning rate schedule.
result SWALP matches full-precision SGD performance with 8-bit quantization and converges to optimal solutions.
ADPSGD optimizes SGD convergence by varying averaging periods based on model variance.
problem High communication costs in parallel SGD.
method Varying averaging period based on model variance.
result ADPSGD achieves better convergence and smaller communication compared to CPSGD.
Averaging SGD weights improves deep learning models.
problem Improving deep learning model generalization.
method Simple averaging of SGD weights with cyclical or constant learning rate.
result SWA leads to better generalization and flatter solutions.
Paper explores how combining tail-averaging and minibatching improves SGD convergence.
problem Understanding and optimizing learning properties of SGD variants.
method Least squares learning in a nonparametric setting, focusing on multiple passes, mini-batching, and averaging.
result Tail averaging allows faster convergence rates than uniform averaging in nonparametric settings.
Partial model averaging improves Federated Learning performance.
problem Periodic model averaging causes significant model discrepancy in Federated Learning.
method Proposes a partial model averaging framework that encourages local models to stay close to each other.
result Partial averaging achieves up to 2.2% higher validation accuracy than full averaging.
This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…
Simplified analysis of SGD for linear regression with weight averaging.
problem Understanding SGD optimization in linear regression models.
method Simplified analysis using linear algebra tools, bypassing complex operator manipulations.
result Recovery of bias and variance bounds for SGD in linear regression.
SGD in linear regression overfits but performs well due to bias-variance trade-off.
problem Understanding overfitting in SGD for linear regression.
method Constant-stepsize SGD with iterate averaging or tail averaging, analyzing full eigenspectrum of data covariance matrix.
result Sharp excess risk bounds revealing bias-variance decomposition for SGD in linear regression.
Unified framework for efficient distributed SGD algorithms.
problem Lack of rigorous convergence analysis and comparative study of communication-reduction strategies.
method Cooperative SGD framework that unifies and analyzes existing communication-efficient SGD algorithms.
result Novel convergence guarantees for existing algorithms and design of new efficient algorithms.
New algorithm reduces communication in distributed SGD, improving efficiency.
problem Slow communication rounds bottleneck synchronous mini-batch SGD convergence.
method Proposes non-asymptotic error analysis for Local-SGD, comparing to averaging methods.
result Local-SGD reduces communication by a factor of O(√T/P^(3/2)) for large step sizes.
A2SGD reduces distributed SGD communication to O(1) per worker.
problem Heavy communication costs in distributed SGD for large models.
method Two-level gradient averaging to consolidate gradients to two local averages.
result Achieves O(1) communication complexity per worker, significantly reducing traffic and training time.
A new distributed SGD algorithm reduces communication by infrequent global reduction.
problem Reducing communication in large-scale machine learning training.
method Hierarchical averaging stochastic gradient descent (Hier-AVG) with infrequent global reduction.
result Hier-AVG achieves comparable training speed with better test accuracy.
Local SGD with periodic averaging achieves faster convergence with less communication.
problem Communication overhead in distributed optimization.
method Local SGD with periodic averaging, Polyak-Łojasiewicz condition, adaptive synchronization.
result Local SGD can achieve linear speed up with fewer communication rounds, especially for non-strongly convex functions.
MDA optimizer performs similarly to SGD+M in CV and Adam in NLP.
problem Performance degradation due to choosing the wrong optimizer.
method Modernized Dual Averaging (MDA) optimizer, inspired by dual averaging.
result MDA performs as well as SGD+M in CV and as Adam in NLP.
Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth object…
Unified analysis of finite weight averaging methods in deep learning.
problem Theoretical explanation of finite weight averaging methods in deep learning.
method Generalized SGD and LAWA, establishing convergence and generalization bounds.
result FWA achieves faster convergence and better generalization compared to SGD.
New insights into using momentum for non-convex optimization.
problem Improving training of non-convex models like deep neural networks.
method Developed a Lyapunov analysis of SGD with momentum using stochastic primal averaging.
result Precise conditions under which SGD+M outperforms SGD and optimal hyper-parameter schedules.
This paper analyzes SGD with increasingly weighted averaging for optimization and generalization.
problem Improving optimization and generalization for non-strongly convex objectives.
method Comprehensive analysis of increasingly weighted averaging schemes for convex, strongly convex, and non-convex objectives.
result The weight α affects both optimization and generalization errors, revealing a trade-off. New bounds for SGD show improved performance in various settings.
problem Improving convergence bounds for SGD with random permutations.
method Analyzing convergence of SGD with random reshuffling and arbitrary permutations.
result Tighter lower bounds for weighted average iterates in both convex and strongly-convex cases.
Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these tw…
New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.
problem The generalization benefits of decentralized learning.
method Proved D-SGD implicitly minimizes SAM's loss function.
result Decentralized SGD and Average-direction SAM are asymptotically equivalent.
The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.
problem Stochastic optimization in a streaming setting with time-dependent and biased gradient estimates.
method Analysis of several first-order methods including SGD, mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages.
result Time-varying mini-batch SGD methods can break long- and short-range dependence structures, and biased SGD methods can achieve comparable performance to their unbiased counterparts.
The paper improves SGD's generalization error bounds for nonconvex optimization.
problem Improving generalization error bounds for SGD in nonconvex optimization.
method Characterizing the on-average stability of SGD iterates and using it to derive probabilistic generalization error bounds.
result Improved generalization error bounds for SGD in both nonconvex and gradient dominant loss functions.
New method averages SGD iterates to achieve adjustable regularization.
problem Overfitting in machine learning models.
method Averaging SGD iterates for regularized solutions.
result Obtain regularized solutions without tuning parameters.
TANGO optimizes models with small learning rates, converging to natural gradient.
problem Optimizing models with small learning rates to converge to natural gradient.
method TANGO, a simple algorithm that converges to natural gradient descent.
result TANGO achieves natural gradient descent with small learning rates.
Consider a number of workers running SGD independently on the same pool of data and averaging the models every once in a while -- a common but not well understood practice. We study model averaging as a variance-reducing mechanism and describe two ways in which the frequency of averaging affects convergence. For convex…
We present a novel method for frequentist statistical inference in M-estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlen…
New framework analyzes SGD dynamics in large samples and dimensions.
problem Analyzing stochastic gradient descent in large-scale settings.
method Inspired by random matrix theory, new framework for fixed stepsize and finite sum settings.
result SGD dynamics become deterministic in the large sample and dimensional limit, governed by a Volterra integral equation.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
SGDM accelerates faster than SGD with large batch sizes and permits broader learning rates.
problem Understanding the role of momentum in SGDM and its convergence rates.
method Analysis of SGDM convergence rates under strongly convex settings, including finite-sample rates and asymptotic normality of the averaged estimator.
result SGDM converges faster than SGD with large batch sizes and permits broader learning rates.
Local SGD outperforms minibatch SGD for quadratic objectives.
problem Theoretical foundations of local SGD are lacking.
method Proved local SGD strictly dominates minibatch SGD for quadratic objectives and accelerated local SGD is minimax optimal.
result Local SGD does not dominate minibatch SGD in general convex objectives.
New stability analysis improves generalization of multipass SGD.
problem Improper preconditioning affects generalization in multipass SGD.
method Developed on-average stability analysis for multipass SGD.
result Proper preconditioning yields optimal effective dimension dependence.
This paper addresses convergence issues in non-convex optimization problems using stagewise learning.
problem Theoretical gaps in convergence for non-convex problems and lack of adaptive step size theories.
method Proposes a stagewise optimization framework for non-smooth non-convex problems using adaptive step sizes.
result Demonstrates adaptive convergence of stagewise AdaGrad and improved generalization performance.
Behavior cloning training instabilities amplified by SGD noise over long horizons.
problem Training instabilities in behavior cloning with deep neural networks.
method Empirical dissection of minibatch SGD updates and their effects on long-horizon rewards.
result Exponential moving average (EMA) of iterates effectively mitigates gradient variance amplification (GVA).
Elastic Gossip distributes neural network training using gossip-like protocols.
problem Distributing neural network training across heterogeneous environments.
method Pairwise-communication using Gossip-like protocols, building on Elastic Averaging SGD.
result Elastic Gossip performs better than Gossiping SGD in experiments, but hyper-parameter search may yield better configurations.
Non-asymptotic rates for SGD via martingale CLT.
problem Improving the convergence rates of SGD.
method Combining Stein's method and Lindeberg's argument for multivariate martingale CLT, then applying to SGD.
result Explicit rates for multivariate martingale CLT and SGD convergence.
AdaComm optimizes SGD by dynamically adjusting communication frequency for faster convergence.
problem Achieving optimal error-runtime trade-off in distributed SGD.
method Adaptive communication strategy that starts with infrequent averaging to save delay and improve speed, then increases frequency.
result AdaComm reduces training time by 3x while maintaining the same final loss.
Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results. (1) We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust …
VR-SGD is a simple method for machine learning that uses larger learning rates and averages.
problem Efficiently solving machine learning problems with large datasets.
method A simple variant of SVRG with specific averaging and update rules.
result VR-SGD achieves linear convergence for strongly convex problems and similar performance to momentum methods for non-strongly convex problems.
The paper provides statistical guarantees for SGD and ASGD in high-dimensional settings.
problem Theoretical understanding of SGD and ASGD in high-dimensional settings.
method Transfer of tools from high-dimensional time series to online learning, using coupling techniques.
result Established geometric-moment contraction and q-th moment convergence of SGD and ASGD. Smoothed SGD improves quantile estimation without crossing curves.
problem Estimating quantiles without crossing estimated curves.
method Smoothed SGD algorithm with Bahadur representation and Gaussian approximation.
result Smoothed SGD provides non-asymptotic tail probability bounds and a Gaussian approximation for quantile estimates.
Sharp bounds established for Federated Averaging (FedAvg), improving convergence rates.
problem Undetermined convergence rate of Federated Averaging (FedAvg) in Federated Learning.
method Developed novel iterate bias concept and proved sharp bounds on it, leading to improved convergence results.
result Lower bounds for FedAvg match existing upper bounds, showing no improvable capacity.
Paper explores using bootstrap methods to improve SGD's stability and robustness.
problem Improving the stability and robustness of SGD.
method Investigates empirical bootstrap approaches for SGD from algorithmic stability and statistical robustness perspectives.
result Demonstrates construction of purely distribution-free confidence intervals using bootstrap SGD.
SGD optimality proven for convex objectives without smoothness assumptions.
problem Proving optimality of SGD for convex objectives without smoothness assumptions.
method Stochastic Gradient Descent (SGD) for convex objectives without smoothness or strict convexity assumptions.
result With high probability, the objective evaluated at the final candidate minimizer is close to the minimal value of the objective.
Paper finds optimal mini-batch size for SGD to speed up learning.
problem Optimizing mini-batch size for faster SGD convergence.
method Empirical inverse law and theoretical bound on mini-batch SGD training.
result An accurate model for predicting training time and identifying implications for algorithm and hardware.