A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New scheme adapts batch size for faster variance-reduced algorithms.
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
Adaptive batch size schedules improve language model training efficiency and generalization.
Adaptive batch sizes improve local gradient methods in distributed training.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
Adaptive SGD learns optimal batch size for strong convex functions.
We propose an adaptive optimization method for deep learning that dynamically adjusts batch size.
This work improves SGD convergence by adaptively adjusting batch sizes.
Adaptive batch sizes improve active learning efficiency and flexibility.
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
For large scale on-line inference problems the update strategy is critical for performance. We derive an adaptive scan Gibbs sampler that optimizes the update frequency by selecting an optimum mini-batch size. We demonstrate performance of our adaptive batch-size Gibbs sampler by comparing it against the collapsed Gibb…
AdaScale SGD adapts learning rates for large-batch training efficiently.
ABS dynamically adjusts batch size based on policy stability, improving RL performance.
The most straightforward method to accelerate Stochastic Gradient Descent (SGD) computation is to distribute the randomly selected batch of inputs over multiple processors. To keep the distributed processors fully utilized requires commensurately growing the batch size. However, large batch training often leads to poor…
New analysis reveals batch size effects on stochastic conditional gradient methods.
In this paper, we propose a novel approach to automatically determine the batch size in stochastic gradient descent methods. The choice of the batch size induces a trade-off between the accuracy of the gradient estimate and the cost in terms of samples of each update. We propose to determine the batch size by optimizin…
New insights into learning rates and batch sizes for neural networks using random matrix theory.
Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…
In this paper, we study the multi-armed bandit problem in the batched setting where the employed policy must split data into a small number of batches. While the minimax regret for the two-armed stochastic bandits has been completely characterized in \cite{perchet2016batched}, the effect of the number of arms on the re…
Improved method finds second-order stationary points privately with better efficiency.
In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to …
Adam's generalization performance is improved by batch size and weight decay in neural networks.
Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (A…
Improved analysis shows momentum in SGD reduces batch size needs for non-convex objectives.
Riemannian stochastic gradient descent converges faster with increasing batch size.
A new scaling law predicts optimal batch size for training models.
A new method SEBS optimizes SGD batch size for better performance.
Equivalent tests for SGD batch size selection found.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
SGD's performance improves with critical batch size, minimizing SFO complexity.
Improved SHB method for faster convergence on strongly-convex quadratics.
In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle poi…
When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a lea…
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.
Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model b…
Adaptive algorithm AMSGrad converges for weakly convex constrained optimization problems.
New algorithm for contextual combinatorial bandits with probabilistic arm triggering.
Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning ra…
In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…
Empirical study on SGD hyperparameters and adversarial robustness.
Standard optimizers perform as well as LARS and LAMB at large batch sizes.
Modern machine learning models are typically trained using Stochastic Gradient Descent (SGD) on massively parallel computing resources such as GPUs. Increasing mini-batch size is a simple and direct way to utilize the parallel computing capacity. For small batch an increase in batch size results in the proportional red…
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, thr…
New method improves model accuracy in Byzantine-robust distributed learning by optimizing batch size.
The paper refines and extends batched kernelized bandits, improving regret bounds and introducing a robust setting.