arXiv research

Paper proposes MABN to stabilize BN in small batch sizes.

CBN improves batch normalization for small mini-batch sizes.

Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footp…

A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend them…

SogCLR uses small batch sizes for global contrastive learning, achieving similar performance to SimCLR.

Batch Normalization (BN) uses mini-batch statistics to normalize the activations during training, introducing dependence between mini-batch elements. This dependency can hurt the performance if the mini-batch size is too small, or if the elements are correlated. Several alternatives, such as Batch Renormalization and G…

New method uses momentum to converge in DC optimization with small batches.

Momentum affects optimization differently at small vs large batch sizes near instability.

A new contrastive learning objective, FlatNCE, fixes resource demands in small-batch training.

Mini-batch stochastic gradient descent (SGD) and variants thereof approximate the objective function's gradient with a small number of training examples, aka the batch size. Small batch sizes require little computation for each model update but can yield high-variance gradient estimates, which poses some challenges for…

Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actual…

Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.

In this paper, we study the multi-armed bandit problem in the batched setting where the employed policy must split data into a small number of batches. While the minimax regret for the two-armed stochastic bandits has been completely characterized in \cite{perchet2016batched}, the effect of the number of arms on the re…

ISAAC Newton uses input-based curvature for efficient training.

Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by d…

Pipelined Backpropagation trains large models without batches efficiently.

Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…

New framework optimizes deep learning training by deferring large batch sizes to late stages.

Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and…

SGD with large learning rates can achieve better test accuracy than expected.

Riemannian stochastic gradient descent converges faster with increasing batch size.

The paper studies batch decompositions of random datasets with probabilistic similarity constraints.

SWAP uses large mini-batches to train DNNs faster with good generalization.

BLAE solves batched linear bandits with optimal regret and practical performance.

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

Deep Neural Networks (DNNs) thrive in recent years in which Batch Normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the reduction operations. In this paper, we propose alleviating this problem through sampling only a small fraction of data for normalization at each i…

Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…

SNGM improves large-batch training accuracy.

New method for identifying best arm in batched multi-armed bandit problems.

In statistical learning for real-world large-scale data problems, one must often resort to "streaming" algorithms which operate sequentially on small batches of data. In this work, we present an analysis of the information-theoretic limits of mini-batch inference in the context of generalized linear models and low-rank…

ABS dynamically adjusts batch size based on policy stability, improving RL performance.

Paper explains contrastive learning using cosine similarity and proposes mitigations for batch size effects.

Large batch sizes reduce gradient variance in DP-SGD, improving privacy.

DReg boosts large-batch SGD's generalization and convergence.

Batch normalization makes deep residual networks train faster.

Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.

This work studies the implicit bias of mini-batch SGD in classification.

BaNk-UCB tackles batched nonparametric bandits with k-NN regression and UCB.

A new method uses extrapolation to train deep learning models with larger batch sizes.

Batching stabilizes risk in high-dimensional linear regression models.

Anytime-valid confirmation of label-shift corrections

Several deep models, esp. the generative, compare the samples from two distributions (e.g. WAE like AutoEncoder models, set-processing deep networks, etc) in their cost functions. Using all these methods one cannot train the model directly taking small size (in extreme -- one element) batches, due to the fact that samp…

This paper studies the inference problem in quantile regression (QR) for a large sample size $n$ but under a limited memory constraint, where the memory can only store a small batch of data of size $m$. A natural method is the naïve divide-and-conquer approach, which splits data into batches of size $m$, computes the l…

A batched Gaussian Process bandit optimization method achieves near-optimal regret bounds.

The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadratic models of the objective function. All of this appears to call for a full batch approach, but sinc…

As an indispensable component, Batch Normalization (BN) has successfully improved the training of deep neural networks (DNNs) with mini-batches, by normalizing the distribution of the internal representation for each hidden layer. However, the effectiveness of BN would diminish with scenario of micro-batch (e.g., less …

We consider two questions at the heart of machine learning; how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? Our work responds to Zhang et al. (2016), who showed deep neural networks can easily memorize randomly labeled training …

We study the problem of training machine learning models incrementally with batches of samples annotated with noisy oracles. We select each batch of samples that are important and also diverse via clustering and importance sampling. More importantly, we incorporate model uncertainty into the sampling probability to com…