arXiv research

New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.

LSAM optimizes deep learning training with improved efficiency.

SGD batch size affects autoencoder global minima sparsity and sharpness.

Momentum affects optimization differently at small vs large batch sizes near instability.

Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which als…

In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It bec…

SGD favors flat minima exponentially more than sharp minima in deep learning.

mSAM improves generalization by making models flatter.

Stochastic gradient descent (SGD) is almost ubiquitously used for training non-convex optimization tasks. Recently, a hypothesis proposed by Keskar et al. [2017] that large batch methods tend to converge to sharp minimizers has received increasing attention. We theoretically justify this hypothesis by providing new pro…

Full-batch GD achieves generalization close to any stationary point with fewer assumptions.

Large-batch stochastic gradient descent (SGD) is widely used for training in distributed deep learning because of its training-time efficiency, however, extremely large-batch SGD leads to poor generalization and easily converges to sharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) from convergin…

New study reveals a polynomial penalty for adapting to unknown margin parameters in batched nonparametric bandits.

New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.

It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding…

Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly und…

In stochastic optimization, using large batch sizes during training can leverage parallel resources to produce faster wall-clock training times per training epoch. However, for both training loss and testing error, recent results analyzing large batch Stochastic Gradient Descent (SGD) have found sharp diminishing retur…

SAM improves neural network generalization by penalizing sharpness, clarifying its exact notion and mechanism.

New findings cast doubt on the role of $λ_{max}$ in generalizing neural networks.

Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.

S-SGD adds symmetrical noise to weights to avoid sharp minima in deep learning.

STORM-PG uses momentum for faster policy gradient updates.

A new method uses extrapolation to train deep learning models with larger batch sizes.

We consider two questions at the heart of machine learning; how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? Our work responds to Zhang et al. (2016), who showed deep neural networks can easily memorize randomly labeled training …

We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…

This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…

This paper studies a decentralized stochastic gradient tracking (DSGT) algorithm for non-convex empirical risk minimization problems over a peer-to-peer network of nodes, which is in sharp contrast to the existing DSGT only for convex problems. To ensure exact convergence and handle the variance among decentralized dat…

Improved theoretical guarantees for SBEED algorithm.

To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly…

Develops a method to estimate policy values robustly in the presence of confounding variables.

Simplified analysis of SGD for linear regression with weight averaging.

Many image processing tasks involve image-to-image mapping, which can be addressed well by fully convolutional networks (FCN) without any heavy preprocessing. Although empirically designing and training FCNs can achieve satisfactory results, reasons for the improvement in performance are slightly ambiguous. Our study i…

The study uses machine learning to predict financial market trends.

Large Sinkhorn couplings improve flow models in data generation tasks.

The overarching goal of this paper is to derive excess risk bounds for learning from exp-concave loss functions in passive and sequential learning settings. Exp-concave loss functions encompass several fundamental problems in machine learning such as squared loss in linear regression, logistic loss in classification, a…

Study shows sample complexity for multicalibration is Θ(ε^-3) with polylogarithmic factors.

Batch normalization (BN) is a technique to normalize activations in intermediate layers of deep neural networks. Its tendency to improve accuracy and speed up training have established BN as a favorite technique in deep learning. Yet, despite its enormous success, there remains little consensus on the exact reason and …

Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good gener…

SGD noise helps select flat minima by concentrating in sharp directions and being proportional to loss value.

New research shows many batch selection methods for training work just as well as full batch training.

Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…

We analyze how batch learning impacts bandit problems.

BatchGFN uses generative flow networks for efficient batch active learning.

Adaptive batch sizes improve active learning efficiency and flexibility.

CBN improves batch normalization for small mini-batch sizes.

Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.

Batch Normalization (BN) uses mini-batch statistics to normalize the activations during training, introducing dependence between mini-batch elements. This dependency can hurt the performance if the mini-batch size is too small, or if the elements are correlated. Several alternatives, such as Batch Renormalization and G…

A new method designs batches for Bayesian optimization more efficiently.

Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…