A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
New framework optimizes deep learning training by deferring large batch sizes to late stages.
Empirical study on SGD hyperparameters and adversarial robustness.
When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a lea…
Adaptive SGD learns optimal batch size for strong convex functions.
Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.
Paper explains contrastive learning using cosine similarity and proposes mitigations for batch size effects.
Adaptive batch sizes improve local gradient methods in distributed training.
New analysis reveals batch size effects on stochastic conditional gradient methods.
Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-siz…
A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend them…
Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally charac…
SGD batch size affects autoencoder global minima sparsity and sharpness.
Neural networks learn the support of the target function through SGD's implicit regularization effect.
Momentum affects optimization differently at small vs large batch sizes near instability.
ABS dynamically adjusts batch size based on policy stability, improving RL performance.
Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size. Previous approaches try to solve this problem by varying the learning rate and batch size over epochs and layers, or some ad hoc modification of the batch normalization. W…
In this paper we study a family of variance reduction methods with randomized batch size---at each step, the algorithm first randomly chooses the batch size and then selects a batch of samples to conduct a variance-reduced stochastic update. We give the linear convergence rate for this framework for composite functions…
In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…
We propose an adaptive optimization method for deep learning that dynamically adjusts batch size.
In this paper, we study the multi-armed bandit problem in the batched setting where the employed policy must split data into a small number of batches. While the minimax regret for the two-armed stochastic bandits has been completely characterized in \cite{perchet2016batched}, the effect of the number of arms on the re…
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we presen…
Muon optimizes training efficiency by improving data retention at large batch sizes.
Adam's generalization performance is improved by batch size and weight decay in neural networks.
New method improves model accuracy in Byzantine-robust distributed learning by optimizing batch size.
Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications o…
We determine the critical batch size for large language models and find it scales with data size, not model size.
New method reduces training time for deep hedging networks.
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
A new scaling law predicts optimal batch size for training models.
Riemannian stochastic gradient descent converges faster with increasing batch size.
With the large rising of complex data, the nonconvex models such as nonconvex loss function and nonconvex regularizer are widely used in machine learning and pattern recognition. In this paper, we propose a class of mini-batch stochastic ADMMs (alternating direction method of multipliers) for solving large-scale noncon…
A new method SEBS optimizes SGD batch size for better performance.
Takagi-Sugeno-Kang (TSK) fuzzy systems are flexible and interpretable machine learning models; however, they may not be easily optimized when the data size is large, and/or the data dimensionality is high. This paper proposes a mini-batch gradient descent (MBGD) based algorithm to efficiently and effectively train TSK …
Noise in SGD helps deep nets generalize better, even with smaller batch sizes.
Adaptive batch size schedules improve language model training efficiency and generalization.
QC-ST and CoCo methods correct batch effects in metabolomics data.
In this study, we consider classification problems based on neural networks in data-imbalanced environment. Learning from an imbalanced data set is one of the most important and practical problems in the field of machine learning. A weighted loss function based on cost-sensitive approach is a well-known effective metho…
SGD's performance improves with critical batch size, minimizing SFO complexity.
Adaptive batch sizes improve active learning efficiency and flexibility.
We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, a…
Modern machine learning models are typically trained using Stochastic Gradient Descent (SGD) on massively parallel computing resources such as GPUs. Increasing mini-batch size is a simple and direct way to utilize the parallel computing capacity. For small batch an increase in batch size results in the proportional red…
Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
SNGM improves large-batch training accuracy.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.