Big batch SGD improves optimization by growing batch sizes to maintain accurate gradients.
problem Noisy gradients in classical SGD make adaptive stepsize selection and automatic stopping difficult.
method Adaptive batch size growth to maintain a constant signal-to-noise ratio in gradient approximations.
result Similar convergence rates to classical SGD, automated learning rate selection possible.
New algorithm reduces communication in distributed SGD, improving efficiency.
problem Slow communication rounds bottleneck synchronous mini-batch SGD convergence.
method Proposes non-asymptotic error analysis for Local-SGD, comparing to averaging methods.
result Local-SGD reduces communication by a factor of O(√T/P^(3/2)) for large step sizes.
Study improves covariance estimation for SGD under Markovian data, matching best rates.
problem Improving covariance estimation for SGD in Markovian data settings.
method Online overlapping batch-means covariance estimator for SGD under Markovian sampling.
result Established convergence rates for covariance estimation under Markovian sampling.
Paper improves SGD for Byzantine failures in distributed optimization.
problem Distributed stochastic optimization with Byzantine failures.
method Variant of SGD that tolerates up to α-fraction of Byzantine machines.
result Achieves approximate minimizers in optimal number of iterations.
Hogwild! adapts to distributed data by varying mini-batch sizes.
problem Efficiently parallelize SGD over distributed local data sets.
method Asynchronous SGD with varying mini-batch sizes, aggregated by an aggregator.
result Improved convergence for heterogeneous data, reducing communication rounds.
DM-SGD uses DPP to diversify mini-batches for SGD, improving model performance.
problem Improving mini-batch diversity in SGD to reduce variance and improve model interpretability.
method DM-SGD uses Determinantal Point Processes (DPP) to select mini-batches with diverse data points.
result DM-SGD outperforms regular SGD and stratified sampling in various setups.
Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
Analysis of SGD+M convergence rates in high dimensions with batch size considerations.
problem Understanding convergence rates of SGD+M in high-dimensional settings.
method Analyzing the dynamics of SGD+M on least squares problems with large batch sizes and dimensions.
result Identifies the implicit conditioning ratio (ICR) that regulates SGD+M's acceleration and convergence rates.
DReg boosts large-batch SGD's generalization and convergence.
problem Large-batch SGD struggles with generalization in deep learning.
method DReg replicates a layer to encourage parameter diversity.
result DReg improves generalization and convergence with large-batch SGD.
A new method SEBS optimizes SGD batch size for better performance.
problem Optimizing batch size for SGD to balance training speed and generalization.
method SEBS method uses a multi-stage geometric batch size enlargement scheme.
result SEBS reduces parameter updates without increasing generalization error.
SNGM improves large-batch training accuracy.
problem Improving generalization in large-batch training.
method Stochastic Normalized Gradient Descent with Momentum.
result SNGM achieves better test accuracy than MSGD and other large-batch methods.
This paper proposes faster machine learning by reducing data access time.
problem Slow training times due to large datasets or feature sizes.
method Systematic and cyclic sampling techniques to reduce data access time.
result Proven to reduce training time up to six times with empirical validation.
Stochastic SGN method converges faster than SGD for DNNs.
problem Training deep neural networks efficiently.
method Stochastic generalized Gauss-Newton method using conjugate gradient and automatic differentiation.
result SGN requires fewer iterations and is more robust to hyperparameters.
Theoretical justification for large batch SGD converging to flatter minima.
problem Understanding the convergence of large batch SGD to flatter minima.
method Theoretical analysis of SGD in finite-time and asymptotic regimes.
result SGD tends to converge to flatter minima regardless of batch size, but with different rates.
SGD's performance improves with critical batch size, minimizing SFO complexity.
problem Optimizing SGD's performance with batch size and learning rate.
method Analysis of SGD using constant and decaying learning rates, focusing on batch size effects.
result SGD with critical batch size minimizes SFO complexity.
Local SGD improves deep learning models' generalization without sacrificing efficiency.
problem Large mini-batches lead to poor model generalization.
method Proposes and evaluates post-local SGD.
result Post-local SGD significantly improves generalization compared to large-batch training.
SGD minima influenced by learning rate, batch size, and gradient covariance.
problem Characterizing the relation between learning rate, batch size, and the properties of SGD minima.
method Approximated SGD by SDE to investigate learning rate, batch size, and gradient covariance effects.
result The ratio of learning rate to batch size is a key determinant of SGD dynamics and minima width, leading to better generalization.
This paper improves Minibatch SGD convergence through typicality sampling.
problem Slow convergence of Minibatch SGD due to large gradient noise.
method Typicality sampling for more efficient batch selection.
result Typical batch SGD outperforms conventional Minibatch SGD in convergence.
SGD batch size affects autoencoder global minima sparsity and sharpness.
problem Investigating how batch size impacts autoencoder learning.
method Non-convex autoencoder training with SGD, varying batch sizes.
result SGD batch size influences global minimum sparsity and sharpness.
Adaptive SGD learns optimal batch size for strong convex functions.
problem Finding optimal batch size for SGD in practice.
method Adaptive SGD method that learns optimal batch size.
result Adaptive SGD exhibits nearly optimal performance in experiments.
This work improves SGD convergence by adaptively adjusting batch sizes.
problem High variance in gradient estimates with small batch sizes.
method Adaptive batch size adjustment based on model training loss.
result Adaptive batch size method requires fewer model updates with same computation.
Paper shows local SGD outperforms mini-batch SGD under certain conditions.
problem Proving local SGD's superiority in distributed learning with heterogeneous data.
method New lower and upper bounds for local SGD under first-order heterogeneity assumptions.
result Local SGD is min-max optimal under certain conditions, resolving understanding of distributed optimization.
SGD converges fast in over-parametrized models that interpolate data.
problem Understanding fast convergence of SGD in modern machine learning.
method Formal analysis of SGD convergence in over-parametrized models.
result SGD converges similarly to full gradient descent in certain regimes.
This work analyzes mini-batching and averaging in SGD for least squares regression.
problem Improving parallelization and reducing variance in SGD for least squares regression.
method Characterizes mini-batching and tail-averaging for least squares regression, providing non-asymptotic excess risk bounds.
result Proves near-linear parallelization speedups and minimax risk for SGD with nearly the same number of updates as batch gradient descent.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.
problem Understanding the stability and convergence of mini-batch SGD.
method Analyzing the mini-batch Hessian and its directional curvature.
result Mini-batch SGD operates in a different stability regime (Edge of Stochastic Stability) compared to full-batch GD.
K-FAC doesn't improve large batch training efficiency.
problem Inefficiency of K-FAC in large batch size training.
method Empirical analysis of K-FAC and SGD on ResNet and AlexNet.
result K-FAC doesn't exhibit improved scalability to large batch sizes.
Empirical study on SGD hyperparameters and adversarial robustness.
problem Effect of SGD hyperparameters on adversarial robustness and generalization.
method Empirical observation of learning rate, batch size, and momentum effects on adversarial robustness and generalization.
result Constant learning rate to batch size ratio leads to good generalization and almost constant adversarial robustness.
GNC smooths loss function for large-batch SGD, improving generalization.
problem Extremely large-batch SGD leads to poor generalization and converges to sharp minima.
method Gradient noise convolution (GNC) smooths loss function by convolving gradient noise with the loss function.
result GNC achieves state-of-the-art generalization performance for large-scale deep neural networks.
Batch augmentation improves deep learning training by reducing batch size requirements.
problem Training deep neural networks with large batches can lead to overfitting.
method Replicate samples within a batch with different data augmentations.
result Batch augmentation reduces the number of necessary SGD updates for achieving the same accuracy.
Wider neural networks perform better with large batches.
problem Communication overheads in small-batch training.
method Theoretical analysis and experiments on neural networks.
result Wider networks are more suitable for large-batch training.
Online learning improves big data accuracy quickly.
problem Heterogeneity in big data analysis.
method Online machine learning for big data.
result Online learning converges quickly to batch accuracy.
Paper finds optimal mini-batch size for SGD to speed up learning.
problem Optimizing mini-batch size for faster SGD convergence.
method Empirical inverse law and theoretical bound on mini-batch SGD training.
result An accurate model for predicting training time and identifying implications for algorithm and hardware.
Stochastic momentum methods trade compute efficiency for serial runtime.
problem Stochastic momentum methods trade compute efficiency for serial runtime.
method Stochastic HB and ASGD for consistent linear regression with Gaussian covariates.
result HB preserves SGD-level CE over a larger batch-size window, allowing larger batches to reduce serial runtime until HB reaches its deterministic accelerated scale.
Generalizes SGD convergence analysis and optimizes stepsize.
problem Analyzes convergence of stochastic gradient descent (SGD) under arbitrary sampling.
method Introduces expected smoothness and derives convergence rates for various mini-batching strategies.
result Derives optimal mini-batch size and stepsize-switching rules.
Neural networks learn the support of the target function through SGD's implicit regularization effect.
problem Learning the support of the target function in neural networks.
method Investigation of mini-batch SGD's ability to learn the support in the first layer of a neural network.
result Mini-batch SGD effectively learns the support in the first layer by shrinking irrelevant weights, while vanilla GD requires an explicit regularization term.
AdaScale SGD adapts learning rates for large-batch training efficiently.
problem Adapting learning rates for large-batch training to balance speed-ups and model quality.
method Adaptive learning rate adaptation based on gradient variance.
result AdaScale achieves reliable speed-ups for a wide range of batch sizes without degrading model quality.
Improves deep learning training by matching mini-batch distributions.
problem Overfitting and noise in mini-batch training.
method ITDM, which matches the moments of mini-batch distributions to reduce overfitting.
result ITDM reduces overfitting and improves DNN training.
New statistical properties for mini-batch Cox-NN optimization.
problem Optimizing deep Cox neural networks using mini-batches.
method Developed mini-batch maximum partial-likelihood estimator (mb-MPLE) for Cox-NN.
result mb-MPLE is consistent and achieves optimal convergence rate.
SGD noise has no bias advantage in online learning, contrary to offline learning.
problem The role of SGD noise in online learning.
method Extensive empirical analysis of image and language data.
result Small batch sizes do not confer any implicit bias advantages in online learning.
The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.
problem Stochastic optimization in a streaming setting with time-dependent and biased gradient estimates.
method Analysis of several first-order methods including SGD, mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages.
result Time-varying mini-batch SGD methods can break long- and short-range dependence structures, and biased SGD methods can achieve comparable performance to their unbiased counterparts.
SGD approximates diffusion processes in nonconvex optimization.
problem Nonconvex optimization problems in machine learning.
method Diffusion approximation of SGD using master equation.
result SGD dynamics can escape local minima and saddle points.
Full-batch GD outperforms one-pass SGD in learning a single-index model with quadratic activation.
problem Learning a single-index model with quadratic activation using gradient descent.
method Full-batch gradient descent compared to one-pass stochastic gradient descent (SGD) on a correlation loss.
result Full-batch GD requires only n≃d samples for strong recovery, while one-pass SGD requires n≳dlogd samples. Nesterov SGD doesn't accelerate over SGD in over-parameterized learning.
problem Theoretical and practical acceleration of SGD with momentum in over-parameterized learning.
method Introducing a compensation term to Nesterov SGD, resulting in MaSS algorithm.
result MaSS converges for same step sizes as SGD and achieves accelerated convergence rates over SGD.
SGD dynamics influence neural network training speed and generalization.
problem Understanding SGD dynamics in relation to neural network loss.
method Empirical investigation and analysis of SGD dynamics in relation to the sharpest directions of the loss.
result Reducing learning rate along the sharpest directions improves training speed and generalization.
This paper investigates factors influencing SGD minima.
problem Understanding the factors that influence the minima found by SGD.
method Examined learning rate, batch size, Hessian, and gradient covariance; used stochastic differential equations to model SGD.
result The ratio of batch size to learning rate is a main factor in SGD dynamics.
Optimal batch size minimizes training time for neural networks.
problem Minimizing training time for two-layer neural networks with SGD.
method Characterized optimal batch size as a function of target hardness (information exponents). Used Correlation loss SGD to overcome limitations.
result Optimal batch size minimizes training time without changing total sample complexity.
EE-Grad optimizes mini-batch sizes for SGD efficiency.
problem Optimizing cost-efficiency in mini-batch SGD.
method Sequential exploration and exploitation of mini-batch oracles.
result EE-Grad achieves optimal cost-efficiency with performance guarantees.