A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Local SGD outperforms minibatch SGD for quadratic objectives.
Stochastic gradient descent~(SGD) and its variants have become more and more popular in machine learning due to their efficiency and effectiveness. To handle large-scale problems, researchers have recently proposed several parallel SGD methods for multicore systems. However, existing parallel SGD methods cannot achieve…
Paper bridges statistical inference for DP-SGD, a privacy-preserving machine learning method.
S-SGD adds symmetrical noise to weights to avoid sharp minima in deep learning.
Untuned SGD converges but with an exponential dependence on smoothness, adaptive methods prevent this.
MindFlayer SGD improves parallel SGD for heterogeneous, random compute times.
Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
Due to its efficiency and ease to implement, stochastic gradient descent (SGD) has been widely used in machine learning. In particular, SGD is one of the most popular optimization methods for distributed learning. Recently, quantized SGD (QSGD), which adopts quantization to reduce the communication cost in SGD-based di…
Stochastic gradient descent (SGD) is a popular stochastic optimization method in machine learning. Traditional parallel SGD algorithms, e.g., SimuParallel SGD, often require all nodes to have the same performance or to consume equal quantities of data. However, these requirements are difficult to satisfy when the paral…
This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…
Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in our paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration o…
Adaptive step-size method improves compressed SGD performance in machine learning.
SGD methods fail to converge to global minimizers in deep neural networks with ReLU activation.
The paper analyzes the variance of different shuffling methods in stochastic gradient descent.
Enhanced ROOT-SGD optimizes stochastic optimization with diminishing stepsizes.
Derives effective continuous dynamics for adaptive SGD methods.
To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple workers is limited by the communication overhead. Two approaches, namely pipelining and gradient spar…
Stochastic Gradient Descent (SGD) has become one of the most popular optimization methods for training machine learning models on massive datasets. However, SGD suffers from two main drawbacks: (i) The noisy gradient updates have high variance, which slows down convergence as the iterates approach the optimum, and (ii)…
Paper shows local SGD outperforms mini-batch SGD under certain conditions.
Paper explores using bootstrap methods to improve SGD's stability and robustness.
Introduces a new stochastic optimization method for deep learning.
Local SGD outperforms conventional methods in LLM training.
New insights into using momentum for non-convex optimization.
We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent~(SGD) methods. To reduce the computation costs pertaining to the projections, we pro…
New method for uncertainty quantification in private SGD.
In this work, we consider the resilience of distributed algorithms based on stochastic gradient descent (SGD) in distributed learning with potentially Byzantine attackers, who could send arbitrary information to the parameter server to disrupt the training process. Toward this end, we propose a new Lipschitz-inspired c…
Cheap methods improve uncertainty in SGD solutions.
We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with …
One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a…
Unified analysis of SGD variants for nonconvex federated optimization.
Stochastic Gradient Descent (SGD) is widely used in machine learning problems to efficiently perform empirical risk minimization, yet, in practice, SGD is known to stall before reaching the actual minimizer of the empirical risk. SGD stalling has often been attributed to its sensitivity to the conditioning of the probl…
MDA optimizer performs similarly to SGD+M in CV and Adam in NLP.
In this paper, we propose a simple variant of the original SVRG, called variance reduced stochastic gradient descent (VR-SGD). Unlike the choices of snapshot and starting points in SVRG and its proximal variant, Prox-SVRG, the two vectors of VR-SGD are set to the average and last iterate of the previous epoch, respecti…
Study reveals convergence properties of SGD with random learning rate.
A new method SEBS optimizes SGD batch size for better performance.
We present a novel method for frequentist statistical inference in -estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlen…
Proposes Stochastic-Sign SGD for federated learning with theoretical guarantees.
RS-NSGD improves SGD convergence for heavy-tailed noise.
In this paper, we propose a simple variant of the original stochastic variance reduction gradient (SVRG), where hereafter we refer to as the variance reduced stochastic gradient descent (VR-SGD). Different from the choices of the snapshot point and starting point in SVRG and its proximal variant, Prox-SVRG, the two vec…
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
HybridSGD improves SGD performance by balancing computation and communication.
DReg boosts large-batch SGD's generalization and convergence.
Machine learning, especially deep neural networks, has been rapidly developed in fields including computer vision, speech recognition and reinforcement learning. Although Mini-batch SGD is one of the most popular stochastic optimization methods in training deep networks, it shows a slow convergence rate due to the larg…
The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it. Existing works based on stability have studied nonconvex loss functions, but only considered the generalization error of the SGD in…
This paper analyzes Local SGD for federated learning, achieving both statistical and communication efficiency.
Balls-and-Bins sampling improves DP-SGD privacy and utility.
SNGM improves large-batch training accuracy.
Proposes a new adaptive learning rate for SGD.