Local SGD outperforms minibatch SGD for quadratic objectives.
problem Theoretical foundations of local SGD are lacking.
method Proved local SGD strictly dominates minibatch SGD for quadratic objectives and accelerated local SGD is minimax optimal.
result Local SGD does not dominate minibatch SGD in general convex objectives.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
It is well known that, for most datasets, the use of large-size minibatches for Stochastic Gradient Descent (SGD) typically leads to slow convergence and poor generalization. On the other hand, large minibatches are of great practical interest as they allow for a better exploitation of modern GPUs. Previous literature …
Study on the noise in SGD minibatches near local minima.
problem Understanding the noise in SGD minibatches near local minima.
method Detailed analysis of SGD noise in linear regression and derivation of a general formula for different types of minima.
result Provides insight into the stability of training neural networks and suggests large learning rates can help generalization.
This work improves SGD minibatch sampling using determinantal point processes based on orthogonal polynomials.
problem Improving variance reduction in stochastic gradient descent (SGD) for large datasets.
method Orthogonal polynomial-based determinantal point processes for sampling minibatches in SGD.
result DPP minibatches lead to a smaller mean square approximation error than uniform minibatches.
Machine learning, especially deep neural networks, has been rapidly developed in fields including computer vision, speech recognition and reinforcement learning. Although Mini-batch SGD is one of the most popular stochastic optimization methods in training deep networks, it shows a slow convergence rate due to the larg…
Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better char…
New bounds for M-SGD show its error distribution is nearly Gaussian.
problem Understanding the error distribution of M-SGD.
method Proved non-asymptotic bounds for M-SGD in Wasserstein distance.
result Error distribution of M-SGD is approximately Gaussian.
Paper proves minibatch SGD for GP inference converges and improves generalization.
problem Theoretical understanding and practical use of SGD for correlated samples in Gaussian process inference.
method Proves minibatch SGD converges to a critical point with rate O(1/K) for K iterations, under certain kernel conditions.
result Minibatch SGD for GP inference improves generalization and reduces computational burden.
SGD converges to global minimum for structured non-convex functions.
problem Optimizing non-convex functions using SGD with slow convergence rates.
method Convergence theorems for SGD on structured non-convex functions, including Quasar and PL conditions.
result SGD converges to global minimum for specific non-convex functions under certain conditions.
New convergence bounds for shuffling-based SGD methods in distributed learning.
problem Analyzing the performance of shuffling-based variants of SGD in distributed learning.
method Study of minibatch and local Random Reshuffling methods, proving convergence bounds and lower bounds.
result Shuffling-based variants converge faster than with-replacement sampling methods, and the bounds are tight.
SGD with large learning rates can converge to local maxima.
problem Understanding the behavior of SGD with large learning rates.
method Constructing worst-case optimization problems.
result SGD can converge to local maxima under certain conditions.
Stochastic Gradient Descent (SGD) is a popular optimization method which has been applied to many important machine learning tasks such as Support Vector Machines and Deep Neural Networks. In order to parallelize SGD, minibatch training is often employed. The standard approach is to uniformly sample a minibatch at each…
DReg boosts large-batch SGD's generalization and convergence.
problem Large-batch SGD struggles with generalization in deep learning.
method DReg replicates a layer to encourage parameter diversity.
result DReg improves generalization and convergence with large-batch SGD.
Differentiable learning via SGD and GD can simulate various learning problems, depending on precision and minibatch size.
problem Understanding the power of differentiable learning via SGD and GD compared to statistical query (SQ) learning.
method Comparing the learning power of SGD and GD on population and empirical losses with statistical query learning.
result The learning power of SGD and GD depends on the precision of gradient calculations relative to the minibatch size or sample size.
The Lookahead optimizer improves SGD's performance and generalization without restrictive assumptions.
problem Improving the generalization of SGD with Lookahead.
method A rigorous stability and generalization analysis of the Lookahead optimizer with minibatch SGD, leveraging on-average model stability.
result Derives generalization bounds for convex and strongly convex problems without the restrictive Lipschitzness assumption, demonstrating a linear speedup with batch size.
SGD efficiently learns the XOR function with near-optimal sample complexity.
problem Learning the XOR function with a 2-layer neural network.
method Minibatch SGD on a 2-layer neural network with ReLU activations, focusing on signal-finding and signal-heavy phases.
result Achieves population error o(1) with dextpolylog(d) samples. Noise enhancement improves generalization in training.
problem Improving generalization in training with controlled noise.
method Noise enhancement method to control SGD noise without changing learning rate or minibatch size.
result Noise enhancement improves generalization for real datasets.
While stochastic gradient descent (SGD) is one of the major workhorses in machine learning, the learning properties of many practically used variants are poorly understood. In this paper, we consider least squares learning in a nonparametric setting and contribute to filling this gap by focusing on the effect and inter…
Doubly SGD improves convergence for intractable objective optimization.
problem Optimizing objectives in sum of intractable expectations.
method Doubly SGD with doubly stochastic gradients and independent minibatching.
result Established convergence of doubly SGD under general conditions, including dependent component gradient estimators.
The paper analyzes how noise geometry influences the performance of SGD in machine learning.
problem Understanding how noise geometry affects the performance of stochastic gradient descent.
method Developed two metrics to quantify noise alignment strength and analyzed their effects on loss and subspace projection dynamics.
result Noise geometry can be used to guarantee alignment under certain conditions, aiding SGD's ability to escape from sharp minima.
Behavior cloning training instabilities amplified by SGD noise over long horizons.
problem Training instabilities in behavior cloning with deep neural networks.
method Empirical dissection of minibatch SGD updates and their effects on long-horizon rewards.
result Exponential moving average (EMA) of iterates effectively mitigates gradient variance amplification (GVA).
Unified analysis of stochastic gradient methods for convex and smooth optimization.
problem Minimizing composite convex and smooth functions.
method Unified convergence analysis of various stochastic gradient methods.
result Unified convergence rates for a variety of methods including proximal SGD, variance reduced methods, quantization, and coordinate descent.
Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data. Previous works assumed eit…
Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size N after O(Nε1) iterations achieves ε−suboptimality in expected quadratic distance. This paper studies communication efficiency in federated learning by optimizing the sum-rate-distortion function for indirect multiterminal source coding.
problem Indirect multiterminal source coding in federated learning where edge devices send noisy gradients to the server.
method Analyzes the rate region for the quadratic vector Gaussian CEO problem under unbiased estimator and derives an explicit formula for the sum-rate-distortion function.
result Derives an explicit formula for the sum-rate-distortion function in the special case of identical gradients over edge devices.
We present a novel Metropolis-Hastings method for large datasets that uses small expected-size minibatches of data. Previous work on reducing the cost of Metropolis-Hastings tests yield variable data consumed per sample, with only constant factor reductions versus using the full dataset for each sample. Here we present…
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
We present an approach towards convex optimization that relies on a novel scheme which converts online adaptive algorithms into offline methods. In the offline optimization setting, our derived methods are shown to obtain favourable adaptive guarantees which depend on the harmonic sum of the queried gradients. We furth…
Noise balance theory explains SGD's behavior in neural networks.
problem Understanding SGD's navigation in neural network loss landscapes.
method Analyzes minibatch noise and loss function symmetries.
result Derives the stationary distribution of SGD for deep networks.
Improved SGD learning for single index models reduces sample complexity.
problem Learning a single index model with optimal sample complexity.
method Using smoothed loss in online SGD to reduce sample complexity.
result Online SGD with smoothed loss achieves optimal sample complexity of dk⋆/2. Study the properties of SGD in non-vanishing learning rate regime.
problem Understanding the noise and fluctuation in SGD with finite learning rates.
method Derive exact solvable results for discrete-time SGD in quadratic loss functions.
result Fluctuation caused by discrete-time dynamics is larger than continuous-time theory predicts.
New method improves convergence of SPP for convex optimization problems.
problem Stochastic optimization and robustness to SGD.
method Minibatch Stochastic Proximal Point (M-SPP) method with stability analysis.
result M-SPP achieves faster convergence rates under smoothness and quadratic growth conditions.
Improved convergence analysis for decentralized non-convex optimization.
problem Minimizing a sum of smooth non-convex functions over a network.
method Gradient tracking in decentralized stochastic gradient descent (GT-DSGD).
result GT-DSGD achieves network-independent performances matching centralized SGD under certain conditions.
Large-scale nonconvex optimization problems are ubiquitous in modern machine learning, and among practitioners interested in solving them, Stochastic Gradient Descent (SGD) reigns supreme. We revisit the analysis of SGD in the nonconvex setting and propose a new variant of the recently introduced expected smoothness as…
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Noise in SGD helps deep nets generalize better, even with smaller batch sizes.
problem The generalization benefit of using noise in SGD over large batch sizes.
method Carefully designed experiments and rigorous hyperparameter sweeps on various models.
result Small or moderately large batch sizes outperform very large batches on test sets.
Exact minibatch MH method improves scalability for large datasets.
problem Inexactness in minibatch MH methods causes inference errors.
method TunaMH proposes an exact minibatch MH method with a tunable batch size.
result TunaMH is asymptotically optimal in terms of batch size.
Gibbs sampling is a Markov chain Monte Carlo method that is often used for learning and inference on graphical models. Minibatching, in which a small random subset of the graph is used at each iteration, can help make Gibbs sampling scale to large graphical models by reducing its computational cost. In this paper, we p…
Study on gradient complexity of private optimization with private oracles.
problem Analyzing the efficiency of differentially private optimization algorithms.
method Lower bounds on the number of first-order oracle queries for private optimization.
result Lower bounds on the number of queries for private optimization algorithms, showing a dimension-dependent runtime penalty.
Recent work has established an empirically successful framework for adapting learning rates for stochastic gradient descent (SGD). This effectively removes all needs for tuning, while automatically reducing learning rates over time on stationary problems, and permitting learning rates to grow appropriately in non-stati…
SGD quickly learns a spurious XOR feature before the signal feature, revealing learning dynamics.
problem Over-reliance on spurious correlations in neural networks trained by SGD.
method Theoretical analysis of SGD on two-layer ReLU networks trained on XOR data.
result SGD learns the spurious feature first and exponentially fast, dominating the signal feature.
New sampling technique improves KGC model performance.
problem Ignoring entity neighbors in minibatches affects KGC model training.
method Random-walk based minibatch sampling.
result Proposed method achieves state-of-the-art performance on DB100K.
We propose a new metaheuristic training scheme that combines Stochastic Gradient Descent (SGD) and Discrete Optimization in an unconventional way. Our idea is to define a discrete neighborhood of the current SGD point containing a number of "potentially good moves" that exploit gradient information, and to search this …
New algorithm reduces FL sample and communication costs.
problem Optimizing FL for minimal samples and rounds.
method Stochastic two-sided momentum algorithm.
result Achieves near-optimal sample and communication complexities.
This paper analyzes minibatch optimal transport distances and their applications.
problem Optimal transport distances are complex and impractical for large datasets.
method Extended analysis of minibatch optimal transport distances, focusing on various kernels and debiased functions.
result Minibatch optimal transport distances are unbiased estimators and have statistical and optimisation properties.
There is currently great interest in applying neural networks to prediction tasks in medicine. It is important for predictive models to be able to use survival data, where each patient has a known follow-up time and event/censoring indicator. This avoids information loss when training the model and enables generation o…
PAGE is a simple gradient estimator for nonconvex optimization problems.
problem Nonconvex optimization problems in machine learning.
method PAGE is a probabilistic gradient estimator that uses vanilla SGD with probability and a small adjustment with probability 1-p.
result PAGE achieves optimal convergence rates for nonconvex finite-sum and online problems.