Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
Stochastic Gradient Descent (SGD) is the most popular algorithm for training deep neural networks (DNNs). As larger networks and datasets cause longer training times, training on distributed systems is common and distributed SGD variants, mainly asynchronous and synchronous SGD, are widely used. Asynchronous SGD is com…
Local SGD outperforms conventional methods in LLM training.
problem Training large language models on distributed devices.
method Local SGD algorithm applied to distributed training.
result Local SGD achieves competitive results in LLM training.
SNGM improves large-batch training accuracy.
problem Improving generalization in large-batch training.
method Stochastic Normalized Gradient Descent with Momentum.
result SNGM achieves better test accuracy than MSGD and other large-batch methods.
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly und…
Stochastic gradient descent (SGD) has been found to be surprisingly effective in training a variety of deep neural networks. However, there is still a lack of understanding on how and why SGD can train these complex networks towards a global minimum. In this study, we establish the convergence of SGD to a global minimu…
Stochastic gradient descent (SGD) is a popular stochastic optimization method in machine learning. Traditional parallel SGD algorithms, e.g., SimuParallel SGD, often require all nodes to have the same performance or to consume equal quantities of data. However, these requirements are difficult to satisfy when the paral…
SGD-trained neural networks generalize well even with adversarial label noise.
problem Generalization of neural networks trained on adversarial label noise.
method Training a one-hidden-layer neural network with SGD on arbitrary width networks.
result SGD-trained networks achieve classification accuracy competitive with the best halfspace over adversarial label noise.
The paper explains spikes in training loss as catapults, improving feature learning and generalization.
problem Understanding and improving the training process of neural networks.
method Analysis of training loss spikes in SGD, empirical evidence of catapults, and demonstration of AGOP alignment.
result Catapults in training loss promote feature learning and better generalization.
Artificial intelligence is revolutionizing our lives at an ever increasing pace. At the heart of this revolution is the recent advancements in deep neural networks (DNN), learning to perform sophisticated, high-level tasks. However, training DNNs requires massive amounts of data and is very computationally intensive. G…
With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are l…
S-SGD adds symmetrical noise to weights to avoid sharp minima in deep learning.
problem SGD does not always converge to a flat minimum, leading to poor generalization.
method Symmetrical weight noise injection in SGD.
result S-SGD outperforms conventional SGD and weight-noise injection methods in large batch training.
ARFF reduces spectral bias in SGD-trained neural networks.
problem Spectral bias in two-layer neural networks.
method Comparison of SGD and ARFF on spectral bias and robustness.
result ARFF yields a closer to zero spectral bias compared to SGD.
Overlap-Local-SGD improves distributed SGD by overlapping communication and computation.
problem High communication delay and node slowdown in distributed SGD.
method Adding an anchor model to synchronize local updates and pull them towards the anchor model.
result Overlap-Local-SGD speeds up distributed training and mitigates straggler effects.
SGD-trained models' disagreement predicts test error.
problem Estimating test error of deep networks.
method Empirical testing and theoretical analysis of SGD ensembles.
result SGD ensembles' disagreement correlates with test error.
DReg boosts large-batch SGD's generalization and convergence.
problem Large-batch SGD struggles with generalization in deep learning.
method DReg replicates a layer to encourage parameter diversity.
result DReg improves generalization and convergence with large-batch SGD.
New bounds for KANs trained with DP-SGD, addressing correlated noise.
problem Risk bounds for Kolmogorov-Arnold Networks trained by DP-SGD with correlated noise.
method Established new optimization and population risk analysis for KANs trained with DP-SGD, addressing correlated noise.
result First optimization and population risk analysis of correlated-noise mechanisms for DP training in non-convex settings, including neural networks.
Optimizes SGD for anytime neural networks, improving accuracy.
problem Training networks that produce increasingly accurate outputs over time.
method Orthogonalized SGD optimizer for nested architectures.
result Significantly improves generalization accuracy of anytime networks.
SGD-trained deep nets have bounds on their generalization error.
problem Bounding generalization error for deep neural networks trained by SGD.
method Combining dynamical control of parameter norms and Rademacher complexity estimates.
result Explicit bounds depend on loss trajectory, work for various architectures.
CSER improves SGD efficiency by resetting errors and partial synchronization.
problem Limited scalability of Distributed Stochastic Gradient Descent (SGD) due to communication bottlenecks.
method Introduces 'error reset' technique and partial synchronization for gradients and models.
result Proves convergence for smooth non-convex problems and accelerates distributed training significantly.
SGD's performance improves with critical batch size, minimizing SFO complexity.
problem Optimizing SGD's performance with batch size and learning rate.
method Analysis of SGD using constant and decaying learning rates, focusing on batch size effects.
result SGD with critical batch size minimizes SFO complexity.
Stochastic Gradient Descent introduces noise in training, affecting model decision boundaries.
problem Understanding the impact of noise in SGD on model decision boundaries.
method Characterized SGD and persistent SGD dynamics in a neural network model, measuring noise magnitude in both under- and over-parametrized regimes.
result Noisier algorithms lead to wider decision boundaries in constraint satisfaction problems.
New insights into SGD and generalization via shift-curvature and bias-curvature mechanisms.
problem Understanding the role of curvature in generalization and how SGD affects it.
method Derivation of new SGD steady-state distribution and analysis of shift-curvature and bias-curvature mechanisms.
result Shift-curvature is a significant factor in test performance, especially for small SGD noise.
SGD improves generalization by using gradient variability as a proxy for data randomness.
problem Improving generalization in machine learning models trained with stochastic gradient descent.
method Bootstrap perspective on SGD, analyzing gradient variability and algorithmic variability.
result SGD avoids spurious solutions and improves generalization by implicitly regularizing the trace of the gradient covariance matrix.
Paper improves differential privacy SGD by considering data heterogeneity.
problem Improving differential privacy in machine learning with varying data contributions.
method Introducing influence function to quantify data contributions, designing PIDP-SGD algorithm.
result PIDP-SGD significantly improves machine learning model performance.
SRSGD improves DNN training speed and accuracy.
problem Training deep neural networks is computationally expensive and slow.
method Scheduled Restart SGD (SRSGD) combines NAG momentum with momentum reset.
result SRSGD achieves better error rates with fewer training epochs.
Noise enhancement improves generalization in training.
problem Improving generalization in training with controlled noise.
method Noise enhancement method to control SGD noise without changing learning rate or minibatch size.
result Noise enhancement improves generalization for real datasets.
Distributing Neural Network training is of particular interest for several reasons including scaling using computing clusters, training at data sources such as IOT devices and edge servers, utilizing underutilized resources across heterogeneous environments, and so on. Most contemporary approaches primarily address sca…
New method trains deep networks robustly without adaptive methods.
problem Training deep networks with robustness and efficiency.
method Scale invariant architecture + SGD + weight decay + gradient clipping.
result SGD can achieve similar performance to adaptive methods like Adam.
Several works have aimed to explain why overparameterized neural networks generalize well when trained by Stochastic Gradient Descent (SGD). The consensus explanation that has emerged credits the randomized nature of SGD for the bias of the training process towards low-complexity models and, thus, for implicit regulari…
To accelerate the training of machine learning models, distributed stochastic gradient descent (SGD) and its variants have been widely adopted, which apply multiple workers in parallel to speed up training. Among them, Local SGD has gained much attention due to its lower communication cost. Nevertheless, when the data …
SGD trains NNs to learn low-dimensional representations efficiently.
problem Training two-layer NNs with SGD for multiple-index models.
method Online SGD with weight decay for training.
result First-layer weights converge to the principal subspace of the true model.
Balls-and-Bins sampling improves DP-SGD privacy and utility.
problem Improving privacy and utility in DP-SGD implementations.
method Introducing Balls-and-Bins sampling as an alternative to shuffling in DP-SGD.
result Balls-and-Bins sampling achieves utility comparable to shuffling while offering better privacy amplification.
Proposes a new adaptive learning rate for SGD.
problem Finding an efficient learning rate for SGD.
method Stochastic Polyak step-size (SPS).
result SGD with SPS converges faster for over-parameterized models.
DP-SGD analysis shows many datapoints leak less privacy than previously thought.
problem Empirical evidence suggests DP-SGD leaks less privacy than current analysis predicts.
method Developed a per-instance DP analysis of DP-SGD, introducing dependence on dataset distribution.
result Formally shows DP-SGD leaks significantly less privacy for many datapoints on common benchmarks.
Label noise in SGD helps converge to flatter minima.
problem Improving generalization in overparametrized models.
method Analyzes SGD with label noise, showing convergence to regularized minima.
result SGD with label noise converges to flatter minima, improving generalization.
Large learning rates cause oscillations in NN weights that improve generalization.
problem Improving generalization of neural networks trained with large learning rates.
method Theoretical analysis and feature-noise data generation model.
result Oscillating SGD with large learning rates benefits NN generalization by effectively learning weak features.
SGD methods fail to converge to global minimizers in deep neural networks with ReLU activation.
problem Failure of SGD methods to converge to global minimizers in deep neural networks.
method Stochastic Gradient Descent (SGD) and its variants like Adam, RMSProp, etc.
result SGD methods fail to converge to global minimizers with high probability in deep neural networks with ReLU activation.
Study predicts SGD test loss for structured features.
problem Understanding test loss dynamics in SGD for structured data.
method Solveable model of SGD on mean square loss for arbitrary covariance structure.
result Simpler Gaussian model accurately predicts test loss of nonlinear models.
Empirical study on SGD hyperparameters and adversarial robustness.
problem Effect of SGD hyperparameters on adversarial robustness and generalization.
method Empirical observation of learning rate, batch size, and momentum effects on adversarial robustness and generalization.
result Constant learning rate to batch size ratio leads to good generalization and almost constant adversarial robustness.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
Differential privacy is a useful tool to build machine learning models which do not release too much information about the training data. We study the Rényi differential privacy of stochastic gradient descent when each training example is sampled without replacement (also known as cyclic SGD). Cyclic SGD is typically f…
SQuARM-SGD improves decentralized SGD efficiency with momentum.
problem Efficient decentralized training of large-scale models over networks.
method Fixed local SGD steps with Nesterov's momentum, sparsified and quantized updates, locally computed triggering criterion.
result Convergence rate matches vanilla SGD, momentum improves test performance.
Cost-effective strategies for training machine learning models on volatile cloud instances.
problem Expensive distributed SGD training due to resource interruptions.
method Cost-effective strategies for exploiting volatile cloud instances (spot instances) in distributed SGD.
result Quantified the impact of instance preemption on SGD convergence and training time.
Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its variants, the sample size in each iteration~(epoch) of training is the same as the size of the full training set. In this paper, we propose a…
Study on adversarial training dynamics in high dimensions using SGD.
problem Analyzing adversarial training of models in high-dimensional settings.
method Deriving deterministic equivalents for SGD iterates under Gaussian mixtures.
result No constant learning rate guarantees monotone descent in adversarial training.
Machine learning has made tremendous progress in recent years, with models matching or even surpassing humans on a series of specialized tasks. One key element behind the progress of machine learning in recent years has been the ability to train machine learning models in large-scale distributed shared-memory and messa…
Large SGD step sizes lead to sparse feature learning in neural networks.
problem Sparse feature learning in neural networks with large step sizes.
method Empirical observations and theoretical analysis of SGD dynamics.
result Large step sizes induce implicit regularization leading to sparse predictors.