A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
Adaptive batch size schedules improve language model training efficiency and generalization.
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
Adaptive batch sizes improve local gradient methods in distributed training.
Adaptive batch sizes improve active learning efficiency and flexibility.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-siz…
Adaptive SGD learns optimal batch size for strong convex functions.
We propose an adaptive optimization method for deep learning that dynamically adjusts batch size.
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
AdaScale SGD adapts learning rates for large-batch training efficiently.
ABS dynamically adjusts batch size based on policy stability, improving RL performance.
Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
In this paper, we study the multi-armed bandit problem in the batched setting where the employed policy must split data into a small number of batches. While the minimax regret for the two-armed stochastic bandits has been completely characterized in \cite{perchet2016batched}, the effect of the number of arms on the re…
Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…
Mini-batch stochastic gradient descent (SGD) and variants thereof approximate the objective function's gradient with a small number of training examples, aka the batch size. Small batch sizes require little computation for each model update but can yield high-variance gradient estimates, which poses some challenges for…
For large scale on-line inference problems the update strategy is critical for performance. We derive an adaptive scan Gibbs sampler that optimizes the update frequency by selecting an optimum mini-batch size. We demonstrate performance of our adaptive batch-size Gibbs sampler by comparing it against the collapsed Gibb…
The most straightforward method to accelerate Stochastic Gradient Descent (SGD) computation is to distribute the randomly selected batch of inputs over multiple processors. To keep the distributed processors fully utilized requires commensurately growing the batch size. However, large batch training often leads to poor…
Enhanced DFO using adaptive batch-based FD estimates.
New analysis reveals batch size effects on stochastic conditional gradient methods.
New insights into learning rates and batch sizes for neural networks using random matrix theory.
In this paper, we propose a novel approach to automatically determine the batch size in stochastic gradient descent methods. The choice of the batch size induces a trade-off between the accuracy of the gradient estimate and the cost in terms of samples of each update. We propose to determine the batch size by optimizin…
In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to …
The paper refines and extends batched kernelized bandits, improving regret bounds and introducing a robust setting.
Improved batched SH algorithm maintains original performance.
Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model b…
Adam's generalization performance is improved by batch size and weight decay in neural networks.
Improved method finds second-order stationary points privately with better efficiency.
Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning ra…
Improved SHB method for faster convergence on strongly-convex quadratics.
In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…
Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (A…
Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. However, the optimizer converges slowly at early epochs and there is a gap between large-batch deep learning optimization heuristics and theoretical underpinnings. In this paper, we propose a no…
New algorithm learns optimal policy with multi-step lookahead information.
A new scaling law predicts optimal batch size for training models.
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
The paper tackles fast rates in batch active learning with pool-based data.
Riemannian stochastic gradient descent converges faster with increasing batch size.
SOBER framework optimizes Bayesian optimization tasks efficiently.
Batched Neural Bandits reduces policy updates in sequential decision-making.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
SGD's performance improves with critical batch size, minimizing SFO complexity.
When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a lea…
Existing research shows that the batch size can seriously affect the performance of stochastic gradient descent~(SGD) based learning, including training speed and generalization ability. A larger batch size typically results in less parameter updates. In distributed training, a larger batch size also results in less fr…
Modern machine learning models are typically trained using Stochastic Gradient Descent (SGD) on massively parallel computing resources such as GPUs. Increasing mini-batch size is a simple and direct way to utilize the parallel computing capacity. For small batch an increase in batch size results in the proportional red…
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.
Empirical study on SGD hyperparameters and adversarial robustness.
Unified framework for high-dimensional online learning with non-divergent error bounds and adaptive gains.