arXiv research

RS-NSGD improves SGD convergence for heavy-tailed noise.

Paper explores whether gradient normalization can replace clipping for SGD in heavy-tailed noise.

Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.

SGD converges to critical points of normalized margin in late-stage training for homogeneous neural networks.

SGD's uncertainty quantified in non-convex learning problems.

Improved analysis shows momentum in SGD reduces batch size needs for non-convex objectives.

New theory explains why normalization is preferred in SGD under heavy-tailed noise.

We provide non-asymptotic convergence rates of the Polyak-Ruppert averaged stochastic gradient descent (SGD) to a normal random vector for a class of twice-differentiable test functions. A crucial intermediate step is proving a non-asymptotic martingale central limit theorem (CLT), i.e., establishing the rates of conve…

This paper provides a framework to analyze stochastic gradient algorithms in a mean squared error (MSE) sense using the asymptotic normality result of the stochastic gradient descent (SGD) iterates. We perform this analysis by taking the asymptotic normality result and applying it to the finite iteration case. Specific…

Batch normalization makes deep neural networks' representations increasingly orthogonal.

Paper proposes an online estimator for covariance matrix of SGD iterates.

We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with …

Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this wo…

SNGM improves large-batch training accuracy.

SGDM accelerates faster than SGD with large batch sizes and permits broader learning rates.

Two new covariance estimators for ROOT-SGD improve statistical inference.

Study reveals dynamics of neural networks with normalization, weight decay, and SGD.

Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well on unseen data even when the neural network has millions of parameters…

We present a novel method for frequentist statistical inference in $M$-estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlen…

Paper develops methods for statistical inference in SGD with infinite variance.

We formulate the problem of neural network optimization as Bayesian filtering, where the observations are the backpropagated gradients. While neural network optimization has previously been studied using natural gradient methods which are closely related to Bayesian inference, they were unable to recover standard optim…

New insights into SGD and SGD-M in high dimensions.

Untuned SGD converges but with an exponential dependence on smoothness, adaptive methods prevent this.

Paper proposes DP-SGD and DP-NSGD for differentially private non-convex optimization.

This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized "spectral complexity": their Lipschitz constant, meaning the product of the spectral norms of the weight matrices, times a certain correction factor. This bound is empirically investigated for…

New method for online statistical inference in contextual bandits using SGD.

This paper operationalizes the Exponential Mechanism using Normalizing Flows for private optimization.

The paper analyzes SGD with dropout regularization in linear models, proving asymptotic properties and providing inference tools.

The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistica…

We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher inform…

The redundancy is widely recognized in Convolutional Neural Networks (CNNs), which enables to remove unimportant filters from convolutional layers so as to slim the network with acceptable performance drop. Inspired by the linear and combinational properties of convolution, we seek to make some filters increasingly clo…

New preconditioners speed up SGD on Lie groups.

A method for efficient statistical inference from online algorithms.

Nonlinear SGD achieves high-probability rates in non-convex optimization with heavy-tailed noise.

Deep learning dynamics exhibit anomalous superdiffusion initially, aiding escape from local minima.

Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of …

Early SGD hyperparameters affect deep neural network training, showing a break-even point.

A new method for statistical inference using SGD under $φ$-mixing data.

REPAIR mitigates variance collapse to enable linear interpolation between SGD solutions.

New method improves convergence of SGD for heavy-tailed noise.

SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.

Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The f…

New statistical properties for mini-batch Cox-NN optimization.

In recent studies, several asymptotic upper bounds on generalization errors on deep neural networks (DNNs) are theoretically derived. These bounds are functions of several norms of weights of the DNNs, such as the Frobenius and spectral norms, and they are computed for weights grouped according to either input and outp…

Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good gener…

A new method to improve deep neural networks using weight rescaling.

Spectral portfolio theory links neural networks to wealth dynamics via SGD weight matrices.

Gradient descent on normalized networks reveals sparsity preferences.