Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “SGD analysis”
New analysis shows Local SGD can achieve error scaling with only fixed number of communications.
problem Speeding up SGD by parallelizing across multiple workers with reduced communication overhead.
method Proposed and analyzed Local SGD method with a fixed number of communications independent of the number of steps.
result Achieves an error scaling as 1/(nT) with only a fixed number of communications (Ω(n)).
The paper analyzes the variance of different shuffling methods in stochastic gradient descent.
problem Understanding the variance of different shuffling methods in stochastic gradient descent.
method Power spectral density analysis to study the noise sequences of stochastic gradients.
result The stationary variances of iterates decrease in the order of SGD, SGD-RR, and SGD-SO.
New insights into using momentum for non-convex optimization.
problem Improving training of non-convex models like deep neural networks.
method Developed a Lyapunov analysis of SGD with momentum using stochastic primal averaging.
result Precise conditions under which SGD+M outperforms SGD and optimal hyper-parameter schedules.
Simplified analysis of SGD for linear regression with weight averaging.
problem Understanding SGD optimization in linear regression models.
method Simplified analysis using linear algebra tools, bypassing complex operator manipulations.
result Recovery of bias and variance bounds for SGD in linear regression.
DP-SGD analysis shows many datapoints leak less privacy than previously thought.
problem Empirical evidence suggests DP-SGD leaks less privacy than current analysis predicts.
method Developed a per-instance DP analysis of DP-SGD, introducing dependence on dataset distribution.
result Formally shows DP-SGD leaks significantly less privacy for many datapoints on common benchmarks.
Unified analysis of SGD variants for nonconvex federated optimization.
problem Performance of stochastic gradient methods in nonconvex optimization.
method Proposed a unified assumption for modeling stochastic gradient second moment, leading to a single convergence analysis for various methods.
result Unified convergence analysis for a wide range of SGD variants and distributed methods.
Unified analysis of asynchronous-SGD algorithms for distributed learning.
problem Analyzing asynchronous-SGD in heterogeneous settings with varying speeds and data distributions.
method Unified convergence theory for non-convex smooth functions, including pure asynchronous SGD and its modifications.
result Unified convergence rates for various asynchronous algorithms, including novel methods.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
Study uses SGD to learn operators in Hilbert spaces with convergence analysis.
problem Learning operators in general Hilbert spaces with SGD.
method Proposes weak and strong regularity conditions for convergence analysis.
result SGD converges to best linear approximation of nonlinear operators.
This paper analyzes convergence of DP-SGD with adaptive quantile clipping.
problem Empirical success of adaptive clipping methods lacks theoretical understanding.
method Comprehensive convergence analysis of SGD with quantile clipping (QC-SGD).
result Establishes theoretical guarantees for DP-QC-SGD, revealing relationships between quantile selection, step size, and convergence.
Paper introduces elastic consistency for distributed SGD, enabling convergence analysis.
problem Training large-scale machine learning models in distributed environments.
method Introduces elastic consistency as a general consistency model for distributed SGD.
result Derives convergence bounds for various distributed SGD methods.
RS-NSGD improves SGD convergence for heavy-tailed noise.
problem Nonconvex optimization with heavy-tailed noise.
method Integrates direction normalization into subspace updates.
result Achieves better oracle complexity than full-dimensional normalized SGD.
Paper bridges statistical inference for DP-SGD, a privacy-preserving machine learning method.
problem Asymptotic statistical inference for Differentially Private Stochastic Gradient Descent (DP-SGD).
method Established asymptotic properties of SGD under randomized subsampling, extended to DP-SGD, proposed methods for constructing valid confidence intervals.
result Valid confidence intervals for DP-SGD output achieve nominal coverage rates while maintaining privacy.
SGD noise has no bias advantage in online learning, contrary to offline learning.
problem The role of SGD noise in online learning.
method Extensive empirical analysis of image and language data.
result Small batch sizes do not confer any implicit bias advantages in online learning.
New analysis shows D-SGD can generalize well regardless of graph connectivity.
problem Improving generalization of D-SGD in decentralized settings.
method Algorithmic stability analysis and optimization-dependent generalization bounds.
result D-SGD can achieve generalization bounds similar to classical SGD, independent of graph connectivity.
Unified analysis for decentralized SGD across various topologies and updates.
problem Analysis of decentralized SGD methods with changing topologies and local updates.
method Unified convergence analysis covering local SGD updates and adaptive network topology.
result Universal convergence rates for smooth problems, interpolating between heterogeneous and iid-data settings.
Last SGD iterate bounds for overparameterized linear regression.
problem Analyzing the last iterate risk bounds of SGD with decaying stepsize for overparameterized linear regression.
method Problem-dependent analysis of last iterate risk bounds of SGD with geometrically decaying stepsize.
result Proved nearly matching upper and lower bounds on the excess risk for last iterate SGD with geometrically decaying stepsize.
Study reveals convergence properties of SGD with random learning rate.
problem Analyzing convergence of SGD with random learning rate in non-convex optimization.
method Introduced Poisson SGD with random learning rate and used stationary distribution analysis.
result Poisson SGD converges to a stationary distribution and finds global minima in non-convex optimization.
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
SGD generalization bounds derived from information theory.
problem Understanding generalization of SGD for non-convex functions.
method Combining information-theoretic bounds with perturbation analysis.
result Upper bounds on SGD's generalization error based on gradient variance and function smoothness.
HA-SME models SGD dynamics with Hessian info for better escaping behaviors.
problem Capturing the escaping behaviors of SGD from stationary points.
method HA-SME, a novel SDE with Hessian info in drift and diffusion.
result HA-SME achieves best approximation error and recovers SGD dynamics for quadratics.
Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscs.LG
Continuous-time analysis shows SGD with noise prefers flat minima.
problem Optimizing neural networks using SGD with noise.
method Continuous-time model for SGD with noise analysis.
result Optimization prefers flat minima in certain noise regimes.
SQuARM-SGD improves decentralized SGD efficiency with momentum.
problem Efficient decentralized training of large-scale models over networks.
method Fixed local SGD steps with Nesterov's momentum, sparsified and quantized updates, locally computed triggering criterion.
result Convergence rate matches vanilla SGD, momentum improves test performance.
Doubly SGD improves convergence for intractable objective optimization.
problem Optimizing objectives in sum of intractable expectations.
method Doubly SGD with doubly stochastic gradients and independent minibatching.
result Established convergence of doubly SGD under general conditions, including dependent component gradient estimators.
UD-SGD analysis shows efficient sampling by a few agents can outperform others.
problem Analyzing convergence speed and sampling strategies in UD-SGD.
method Asymptotic analysis of UD-SGD with various communication patterns and sampling strategies.
result Efficient sampling by a few agents can lead to better overall convergence.
This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…
Stochastic Gradient Descent (SGD) is an important algorithm in machine learning. With constant learning rates, it is a stochastic process that, after an initial phase of convergence, generates samples from a stationary distribution. We show that SGD with constant rates can be effectively used as an approximate posterio…
New bounds for KANs trained with DP-SGD, addressing correlated noise.
problem Risk bounds for Kolmogorov-Arnold Networks trained by DP-SGD with correlated noise.
method Established new optimization and population risk analysis for KANs trained with DP-SGD, addressing correlated noise.
result First optimization and population risk analysis of correlated-noise mechanisms for DP training in non-convex settings, including neural networks.
Paper analyzes D-SGD convergence with heterogeneous data and proposes topology learning.
problem Efficiently dealing with data heterogeneity in decentralized learning.
method Revisits D-SGD analysis, introduces neighborhood heterogeneity, and proposes topology learning.
result Formulates topology learning as a tractable optimization problem and demonstrates its effectiveness.
Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning problems. Researchers and industry have developed several techniques to optimize SGD's runtime performance, including asynchronous execution and reduced precision. Our main result is a martingale-based analysis that enables us…
This work analyzes DP-SGD for online LDP problems with practical convergence rates.
problem Analyzing DP-SGD for online LDP problems with practical convergence rates.
method Developed a general framework for online LDP model in stochastic optimization problems, conducted non-asymptotic convergence analysis.
result Comprehensive non-asymptotic convergence analysis of the proposed estimators in finite-sample situations.
New analysis of SGD with MCMC gradient estimator shows convergence rate and saddle point escape.
problem Analyzing SGD with MCMC gradient estimator under complex conditions.
method Introduced MCMC-SGD, analyzed convergence rate and saddle point escape using Bernstein inequality.
result Proven first order convergence rate and saddle point escape at least steps.
SGD with large learning rates can converge to local maxima.
problem Understanding the behavior of SGD with large learning rates.
method Constructing worst-case optimization problems.
result SGD can converge to local maxima under certain conditions.
Stochastic gradient descent (SGD) is the optimization algorithm of choice in many machine learning applications such as regularized empirical risk minimization and training deep neural networks. The classical convergence analysis of SGD is carried out under the assumption that the norm of the stochastic gradient is uni…
ASGD outperforms SGD in overparameterized linear regression, especially in subspaces of small eigenvalues.
problem Generalization of ASGD for overparameterized linear regression.
method Established instance-dependent excess risk bound for ASGD in each eigen-subspace of the data covariance matrix.
result ASGD outperforms SGD in subspaces of small eigenvalues, exhibiting faster decay of bias error.
The paper analyzes how hyperparameters affect SGD with momentum's convergence rate.
problem The role of hyperparameters in SGD with momentum's convergence rate.
method Theoretical analysis using a hyperparameters-dependent stochastic differential equation (hp-dependent SDE).
result The optimal linear rate of convergence depends on both the learning rate and the momentum coefficient.
Paper analyzes high probability convergence of adaptive SGD with momentum.
problem Theoretical understanding of adaptive SGD with momentum in nonconvex settings is incomplete.
method High probability analysis under weak assumptions.
result First high probability convergence proof for gradients to zero in Delayed AdaGrad with momentum.
New bounds for M-SGD show its error distribution is nearly Gaussian.
problem Understanding the error distribution of M-SGD.
method Proved non-asymptotic bounds for M-SGD in Wasserstein distance.
result Error distribution of M-SGD is approximately Gaussian.
New SGD analysis for nonconvex optimization finds optimal rates.
problem Optimizing nonconvex functions in machine learning.
method Introduces a new expected smoothness assumption and analyzes SGD convergence rates.
result Optimal convergence rates for nonconvex smooth functions and global solutions under certain conditions.
MindFlayer SGD improves parallel SGD for heterogeneous, random compute times.
problem Minimizing nonconvex functions with heterogeneous, random compute times.
method MindFlayer SGD, designed for stochastic and heterogeneous delays.
result MindFlayer SGD outperforms existing methods in environments with heavy-tailed noise.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
A distributed SGD method for heterogeneous networks with hubs and workers.
problem Learning in heterogeneous multi-level networks with worker heterogeneity and varying communication.
method Multi-Level Local SGD: distributed SGD with hub-and-spoke paradigm and hub averaging.
result The method converges with error dependent on worker heterogeneity, hub network topology, and iterations.
Enhanced ROOT-SGD optimizes stochastic optimization with diminishing stepsizes.
problem Improving statistical efficiency in stochastic optimization.
method Integrates a diminishing stepsize strategy into ROOT-SGD.
result Achieves optimal convergence rates with improved stability and precision.
The Lookahead optimizer improves SGD's performance and generalization without restrictive assumptions.
problem Improving the generalization of SGD with Lookahead.
method A rigorous stability and generalization analysis of the Lookahead optimizer with minibatch SGD, leveraging on-average model stability.
result Derives generalization bounds for convex and strongly convex problems without the restrictive Lipschitzness assumption, demonstrating a linear speedup with batch size.
Paper analyzes how EMA improves SGD in linear regression.
problem Understanding the effectiveness of EMA in training deep learning models.
method Established risk bound for online SGD with EMA in linear regression.
result SGD with EMA has smaller variance error and exponentially decaying bias error.
Improved analysis shows momentum in SGD reduces batch size needs for non-convex objectives.
problem Reducing batch size requirements for SGD in non-convex optimization.
method Normalized SGD with momentum, adaptive method for small gradient variance.
result Normalized SGD with momentum achieves -critical points in iterations.
An ADRC-incorporated SGD algorithm improves latent factor analysis speed and accuracy.
problem Slow convergence in standard SGD for HDI matrix analysis.
method Incorporates ADRC principles to refine historical and future learning error states.
result Empirically outperforms state-of-the-art LFA models in HDI matrix prediction.