Researchers analyze SGD dynamics using von Mises-Fisher distributions.
problem Understanding the dynamics of stochastic gradient descent in high-dimensional spaces.
method Geometric analysis of minibatch gradient norms and directions through von Mises-Fisher distribution.
result Directional uniformity of minibatch gradients increases over SGD iterations.
This paper improves Minibatch SGD convergence through typicality sampling.
problem Slow convergence of Minibatch SGD due to large gradient noise.
method Typicality sampling for more efficient batch selection.
result Typical batch SGD outperforms conventional Minibatch SGD in convergence.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size N after O(Nε1) iterations achieves ε−suboptimality in expected quadratic distance. This work improves SGD minibatch sampling using determinantal point processes based on orthogonal polynomials.
problem Improving variance reduction in stochastic gradient descent (SGD) for large datasets.
method Orthogonal polynomial-based determinantal point processes for sampling minibatches in SGD.
result DPP minibatches lead to a smaller mean square approximation error than uniform minibatches.
New minibatching strategy reduces stochastic gradient bias in optimisation.
problem Reducing bias in stochastic gradient descent.
method Symmetric Minibatching Strategy combined with momentum.
result Reduced stochastic gradient bias from O(h2) to O(h4). This paper analyzes minibatch optimal transport distances and their applications.
problem Optimal transport distances are complex and impractical for large datasets.
method Extended analysis of minibatch optimal transport distances, focusing on various kernels and debiased functions.
result Minibatch optimal transport distances are unbiased estimators and have statistical and optimisation properties.
Paper analyzes minibatch Wasserstein for machine learning applications.
problem Optimal transport distances on large datasets.
method Analysis of minibatch optimal transport.
result Minibatch Wasserstein is equivalent to implicit regularization with desirable properties.
Paper proves minibatch SGD for GP inference converges and improves generalization.
problem Theoretical understanding and practical use of SGD for correlated samples in Gaussian process inference.
method Proves minibatch SGD converges to a critical point with rate O(1/K) for K iterations, under certain kernel conditions.
result Minibatch SGD for GP inference improves generalization and reduces computational burden.
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
It is well known that, for most datasets, the use of large-size minibatches for Stochastic Gradient Descent (SGD) typically leads to slow convergence and poor generalization. On the other hand, large minibatches are of great practical interest as they allow for a better exploitation of modern GPUs. Previous literature …
Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima. In this work we develop a similar correspondence for minibatch natural gradient descent (NGD). We prove that for sufficiently small learning rates, if the model predictions on the trai…
Anytime MiniBatch speeds up online distributed optimization by handling slow nodes.
problem Mitigating the impact of slow nodes (stragglers) in distributed optimization.
method Proposes an online distributed optimization method that averages minibatch gradients via consensus rounds.
result Prevents stragglers from slowing progress without wasting work.
Stochastic Gradient Descent (SGD) is a popular optimization method which has been applied to many important machine learning tasks such as Support Vector Machines and Deep Neural Networks. In order to parallelize SGD, minibatch training is often employed. The standard approach is to uniformly sample a minibatch at each…
Paper explores how combining tail-averaging and minibatching improves SGD convergence.
problem Understanding and optimizing learning properties of SGD variants.
method Least squares learning in a nonparametric setting, focusing on multiple passes, mini-batching, and averaging.
result Tail averaging allows faster convergence rates than uniform averaging in nonparametric settings.
Unified analysis of stochastic gradient methods for convex and smooth optimization.
problem Minimizing composite convex and smooth functions.
method Unified convergence analysis of various stochastic gradient methods.
result Unified convergence rates for a variety of methods including proximal SGD, variance reduced methods, quantization, and coordinate descent.
New sampling technique improves KGC model performance.
problem Ignoring entity neighbors in minibatches affects KGC model training.
method Random-walk based minibatch sampling.
result Proposed method achieves state-of-the-art performance on DB100K.
SGLRW improves robustness of stochastic gradient MCMC methods.
problem Sensitivity to minibatch size and gradient noise in stochastic-gradient MCMC methods.
method Proposes Stochastic Gradient Lattice Random Walk (SGLRW) with lattice-based discretization.
result SGLRW remains stable in regimes where SGLD fails, including heavy-tailed gradient noise.
New bounds for M-SGD show its error distribution is nearly Gaussian.
problem Understanding the error distribution of M-SGD.
method Proved non-asymptotic bounds for M-SGD in Wasserstein distance.
result Error distribution of M-SGD is approximately Gaussian.
Unbalanced minibatch Optimal Transport improves domain adaptation performance.
problem Optimal transport distances are computationally expensive for large datasets.
method Use unbalanced minibatch Optimal Transport to estimate distances over subsets of data.
result Unbalanced Optimal Transport leads to better domain adaptation results.
A new method for distributed optimization reduces communication rounds without minibatches.
problem Efficient training in distributed machine learning with different data distributions.
method A primal-dual method (GA-MSGD) applied to the Lagrangian of distributed optimization.
result Achieves linear convergence in communication rounds for strongly convex objectives.
New convergence bounds for shuffling-based SGD methods in distributed learning.
problem Analyzing the performance of shuffling-based variants of SGD in distributed learning.
method Study of minibatch and local Random Reshuffling methods, proving convergence bounds and lower bounds.
result Shuffling-based variants converge faster than with-replacement sampling methods, and the bounds are tight.
New technique trains deep neural networks without normalization or minibatch statistics.
problem Training deep neural networks at high learning rates without normalization.
method Channel-wise zero-mean initialization and gradient modification to maintain common mode rejection.
result Achieves higher accuracy compared to batch normalization and shows minibatches are unnecessary.
Study on the noise in SGD minibatches near local minima.
problem Understanding the noise in SGD minibatches near local minima.
method Detailed analysis of SGD noise in linear regression and derivation of a general formula for different types of minima.
result Provides insight into the stability of training neural networks and suggests large learning rates can help generalization.
Paper proposes a method to estimate variance reduction in DNN training using importance sampling.
problem Challenges in assessing variance reduction during DNN training using importance sampling.
method Proposes a method for estimating variance reduction using minibatches sampled under importance sampling.
result Demonstrates consistent reduction in variance, improved training efficiency, and enhanced model accuracy.
Study on gradient complexity of private optimization with private oracles.
problem Analyzing the efficiency of differentially private optimization algorithms.
method Lower bounds on the number of first-order oracle queries for private optimization.
result Lower bounds on the number of queries for private optimization algorithms, showing a dimension-dependent runtime penalty.
Differentiable learning via SGD and GD can simulate various learning problems, depending on precision and minibatch size.
problem Understanding the power of differentiable learning via SGD and GD compared to statistical query (SQ) learning.
method Comparing the learning power of SGD and GD on population and empirical losses with statistical query learning.
result The learning power of SGD and GD depends on the precision of gradient calculations relative to the minibatch size or sample size.
New approach speeds up optimization with repeated gradient steps on same batch.
problem Performance bottlenecks in massive parallel pipelines with large batch sizes.
method Data echoing, taking repeated gradient steps on the same batch.
result Data echoing affords speedups on curvature-dominated part of convergence rate.
We present an approach towards convex optimization that relies on a novel scheme which converts online adaptive algorithms into offline methods. In the offline optimization setting, our derived methods are shown to obtain favourable adaptive guarantees which depend on the harmonic sum of the queried gradients. We furth…
In this paper, we design and analyze a new zeroth-order online algorithm, namely, the zeroth-order online alternating direction method of multipliers (ZOO-ADMM), which enjoys dual advantages of being gradient-free operation and employing the ADMM to accommodate complex structured regularizers. Compared to the first-ord…
This paper studies communication efficiency in federated learning by optimizing the sum-rate-distortion function for indirect multiterminal source coding.
problem Indirect multiterminal source coding in federated learning where edge devices send noisy gradients to the server.
method Analyzes the rate region for the quadratic vector Gaussian CEO problem under unbiased estimator and derives an explicit formula for the sum-rate-distortion function.
result Derives an explicit formula for the sum-rate-distortion function in the special case of identical gradients over edge devices.
Paper examines the structure of stochastic gradients in deep learning.
problem Exploring the structure and heavy tails of stochastic gradients in deep learning.
method Conducted formal statistical tests on stochastic gradients and gradient noise.
result Stochastic gradients and gradient noise do not exhibit power-law heavy tails, but their covariance spectra do.
DP-PCA improves privacy in PCA computations with optimal statistical error.
problem Differentially private principal component analysis with sub-linear sample complexity.
method Private minibatch gradient ascent with private mean estimation.
result Achieves optimal statistical error rates for sub-Gaussian data with n=ildeO(d) samples. We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex. Surprisingly, unlike the smooth case, our knowledge of this fundamental problem is very limited. For example, it is not known whether the proximal stochastic gra…
Local SGD outperforms minibatch SGD for quadratic objectives.
problem Theoretical foundations of local SGD are lacking.
method Proved local SGD strictly dominates minibatch SGD for quadratic objectives and accelerated local SGD is minimax optimal.
result Local SGD does not dominate minibatch SGD in general convex objectives.
Stochastic gradient Markov Chain Monte Carlo (SG-MCMC) has been developed as a flexible family of scalable Bayesian sampling algorithms. However, there has been little theoretical analysis of the impact of minibatch size to the algorithm's convergence rate. In this paper, we prove that under a limited computational bud…
The paper analyzes how noise geometry influences the performance of SGD in machine learning.
problem Understanding how noise geometry affects the performance of stochastic gradient descent.
method Developed two metrics to quantify noise alignment strength and analyzed their effects on loss and subspace projection dynamics.
result Noise geometry can be used to guarantee alignment under certain conditions, aiding SGD's ability to escape from sharp minima.
Exact minibatch MH method improves scalability for large datasets.
problem Inexactness in minibatch MH methods causes inference errors.
method TunaMH proposes an exact minibatch MH method with a tunable batch size.
result TunaMH is asymptotically optimal in terms of batch size.
Improved algorithm reduces stochastic gradient complexity for large-scale learning problems.
problem High stochastic gradient complexity for large-scale learning problems.
method Hybrid Stochastic-Deterministic Minibatch Proximal Gradient (HSDMPG) algorithm.
result Achieves nearly optimal generalization in less than a single pass over data.
SGD with large learning rates can converge to local maxima.
problem Understanding the behavior of SGD with large learning rates.
method Constructing worst-case optimization problems.
result SGD can converge to local maxima under certain conditions.
New method speeds up Gibbs sampling for large graphs.
problem Efficiently sampling from large graphical models.
method Poisson-minibatching Gibbs sampling.
result Theoretical convergence rate guarantees for Poisson-minibatching Gibbs.
Doubly SGD improves convergence for intractable objective optimization.
problem Optimizing objectives in sum of intractable expectations.
method Doubly SGD with doubly stochastic gradients and independent minibatching.
result Established convergence of doubly SGD under general conditions, including dependent component gradient estimators.
SGD converges to global minimum for structured non-convex functions.
problem Optimizing non-convex functions using SGD with slow convergence rates.
method Convergence theorems for SGD on structured non-convex functions, including Quasar and PL conditions.
result SGD converges to global minimum for specific non-convex functions under certain conditions.
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise th…
New method improves convergence of SPP for convex optimization problems.
problem Stochastic optimization and robustness to SGD.
method Minibatch Stochastic Proximal Point (M-SPP) method with stability analysis.
result M-SPP achieves faster convergence rates under smoothness and quadratic growth conditions.
Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data. Previous works assumed eit…
Noise enhancement improves generalization in training.
problem Improving generalization in training with controlled noise.
method Noise enhancement method to control SGD noise without changing learning rate or minibatch size.
result Noise enhancement improves generalization for real datasets.
We analyze a fast incremental aggregated gradient method for optimizing nonconvex problems of the form minx∑ifi(x). Specifically, we analyze the SAGA algorithm within an Incremental First-order Oracle framework, and show that it converges to a stationary point provably faster than both gradient descent and s…