arXiv research

A new method normalizes activations to match batch normalization without batch dependence.

ICAL improves deep learning model accuracy and NLL with optimized batch labeling.

A batched Gaussian Process bandit optimization method achieves near-optimal regret bounds.

New research shows many batch selection methods for training work just as well as full batch training.

Differentiable sorting and rank normalization are incompatible, with specific conditions for admissibility.

Paper improves CMAB regret bounds by reducing batch-size dependency.

Paper tackles robust batched bandits for heavy-tailed rewards.

A new method for batch prediction sets in classification problems.

In this work, we propose a novel technique to boost training efficiency of a neural network. Our work is based on an excellent idea that whitening the inputs of neural networks can achieve a fast convergence speed. Given the well-known fact that independent components must be whitened, we introduce a novel Independent-…

Introduce a variance-weighted batch distribution for diverse sampling in diffusion models.

SWAP uses large mini-batches to train DNNs faster with good generalization.

Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limi…

The paper studies batch decompositions of random datasets with probabilistic similarity constraints.

Federated multi-mini-batch improves efficiency in non-IID environments.

Batch Normalization (BN) uses mini-batch statistics to normalize the activations during training, introducing dependence between mini-batch elements. This dependency can hurt the performance if the mini-batch size is too small, or if the elements are correlated. Several alternatives, such as Batch Renormalization and G…

New constraints on space and adaptivity in bandits force more batches and memory use.

Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.

The paper refines and extends batched kernelized bandits, improving regret bounds and introducing a robust setting.

BYOL learns useful representations without batch statistics.

New test detects independence in streaming data, adapting to data complexity.

SGD batch size affects autoencoder global minima sparsity and sharpness.

New bounds for KANs trained with DP-SGD, addressing correlated noise.

We consider the combinatorial multi-armed bandit (CMAB) problem, where the reward function is nonlinear. In this setting, the agent chooses a batch of arms on each round and receives feedback from each arm of the batch. The reward that the agent aims to maximize is a function of the selected arms and their expectations…

Empirical study on SGD hyperparameters and adversarial robustness.

In this work we investigate the reasons why Batch Normalization (BN) improves the generalization performance of deep networks. We argue that one major reason, distinguishing it from data-independent normalization methods, is randomness of batch statistics. This randomness appears in the parameters rather than in activa…

Develops methods for reliable inference on batched bandit data.

We deal with the efficient parallelization of Bayesian global optimization algorithms, and more specifically of those based on the expected improvement criterion and its variants. A closed form formula relying on multivariate Gaussian cumulative distribution functions is established for a generalized version of the mul…

The accuracy of deep neural networks is significantly affected by how well mini-batches are constructed during the training step. In this paper, we propose a novel adaptive batch selection algorithm called Recency Bias that exploits the uncertain samples predicted inconsistently in recent iterations. The historical lab…

BN refines local partition geometry in piecewise-affine networks during training.

Proposes a new semi-parametric framework for batched bandits with covariates.

Researchers enhance EfficientNet models for practical efficiency on Graphcore IPU.

We propose a general yet simple theorem describing the convergence of SGD under the arbitrary sampling paradigm. Our theorem describes the convergence of an infinite array of variants of SGD, each of which is associated with a specific probability law governing the data selection rule used to form mini-batches. This is…

A new method improves data representation for diverse tasks.

(Mini-batch) Stochastic Gradient Descent is a popular optimization method which has been applied to many machine learning applications. But a rather high variance introduced by the stochastic gradient in each step may slow down the convergence. In this paper, we propose the antithetic sampling strategy to reduce the va…

Can one parallelize complex exploration exploitation tradeoffs? As an example, consider the problem of optimal high-throughput experimental design, where we wish to sequentially design batches of experiments in order to simultaneously learn a surrogate function mapping stimulus to response and identify the maximum of t…

We address the problem of estimating statistics of hidden units in a neural network using a method of analytic moment propagation. These statistics are useful for approximate whitening of the inputs in front of saturating non-linearities such as a sigmoid function. This is important for initialization of training and f…

In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…

New method improves model accuracy in Byzantine-robust distributed learning by optimizing batch size.

Paper addresses OPE for dependent bandit samples using MDS and batch updates.

New analysis for black-box learning without gradients, improving generalization bounds.

A new contrastive learning objective, FlatNCE, fixes resource demands in small-batch training.

Proposes FMC for fair clustering with independent parameters.

Improved deep probabilistic time series forecasting by learning error autocorrelation.

Improved robustness in optimization methods using second-order information.

New hyperparameter ensembles boost neural network performance and uncertainty.

Optimizes RTB bidding without exploration, improving performance under various budgets.

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …

We determine the critical batch size for large language models and find it scales with data size, not model size.