A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper improves CMAB regret bounds by reducing batch-size dependency.
Empirical study on SGD hyperparameters and adversarial robustness.
SGD batch size affects autoencoder global minima sparsity and sharpness.
New method improves model accuracy in Byzantine-robust distributed learning by optimizing batch size.
We determine the critical batch size for large language models and find it scales with data size, not model size.
Batch Normalization (BN) uses mini-batch statistics to normalize the activations during training, introducing dependence between mini-batch elements. This dependency can hurt the performance if the mini-batch size is too small, or if the elements are correlated. Several alternatives, such as Batch Renormalization and G…
We propose a general yet simple theorem describing the convergence of SGD under the arbitrary sampling paradigm. Our theorem describes the convergence of an infinite array of variants of SGD, each of which is associated with a specific probability law governing the data selection rule used to form mini-batches. This is…
Improved robustness in optimization methods using second-order information.
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
We consider the combinatorial multi-armed bandit (CMAB) problem, where the reward function is nonlinear. In this setting, the agent chooses a batch of arms on each round and receives feedback from each arm of the batch. The reward that the agent aims to maximize is a function of the selected arms and their expectations…
Riemannian stochastic gradient descent converges faster with increasing batch size.
A new scaling law predicts optimal batch size for training models.
A new method SEBS optimizes SGD batch size for better performance.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
Adaptive batch size schedules improve language model training efficiency and generalization.
SGD's performance improves with critical batch size, minimizing SFO complexity.
When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a lea…
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.
New analysis reveals batch size effects on stochastic conditional gradient methods.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…
Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
Adaptive SGD learns optimal batch size for strong convex functions.
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
Adaptive batch sizes improve local gradient methods in distributed training.
New analysis for black-box learning without gradients, improving generalization bounds.
Standard optimizers perform as well as LARS and LAMB at large batch sizes.
Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-siz…
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, thr…
Adaptive batch sizes improve active learning efficiency and flexibility.
We propose an adaptive optimization method for deep learning that dynamically adjusts batch size.
In this paper, we propose a novel approach to automatically determine the batch size in stochastic gradient descent methods. The choice of the batch size induces a trade-off between the accuracy of the gradient estimate and the cost in terms of samples of each update. We propose to determine the batch size by optimizin…
In this paper we study a family of variance reduction methods with randomized batch size---at each step, the algorithm first randomly chooses the batch size and then selects a batch of samples to conduct a variance-reduced stochastic update. We give the linear convergence rate for this framework for composite functions…
We investigate the dynamical and convergent properties of stochastic gradient descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the relation between learning rate, batch size and the properties of the final minima, such as width or generalization, remains an open question. In order to tackle this pro…
Increasing the mini-batch size for stochastic gradient descent offers significant opportunities to reduce wall-clock training time, but there are a variety of theoretical and systems challenges that impede the widespread success of this technique. We investigate these issues, with an emphasis on time to convergence and…
Optimal batch size minimizes training time for neural networks.
This paper explores the limits of large batch sizes in deep learning.
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
New oracles improve stochastic optimization with noisy or biased measurements.
SP-NGD improves deep learning models' generalization with large mini-batch sizes.
Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footp…
This work provides a scaling rule for model EMA optimization across batch sizes.
In stochastic optimization, using large batch sizes during training can leverage parallel resources to produce faster wall-clock training times per training epoch. However, for both training loss and testing error, recent results analyzing large batch Stochastic Gradient Descent (SGD) have found sharp diminishing retur…
Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.
This paper studies a decentralized stochastic gradient tracking (DSGT) algorithm for non-convex empirical risk minimization problems over a peer-to-peer network of nodes, which is in sharp contrast to the existing DSGT only for convex problems. To ensure exact convergence and handle the variance among decentralized dat…
Analysis of SGD+M convergence rates in high dimensions with batch size considerations.
Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally charac…