A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
A new contrastive learning objective, FlatNCE, fixes resource demands in small-batch training.
ISAAC Newton uses input-based curvature for efficient training.
Pipelined Backpropagation trains large models without batches efficiently.
Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footp…
We present novel empirical observations regarding how stochastic gradient descent (SGD) navigates the loss landscape of over-parametrized deep neural networks (DNNs). These observations expose the qualitatively different roles of learning rate and batch-size in DNN optimization and generalization. Specifically we study…
SogCLR uses small batch sizes for global contrastive learning, achieving similar performance to SimCLR.
New method uses momentum to converge in DC optimization with small batches.
Anytime-valid confirmation of label-shift corrections
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and…
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
RAD estimates gradients with less memory, faster than small batch sizes.
Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (A…
We propose and evaluate new techniques for compressing and speeding up dense matrix multiplications as found in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, we introduce and study a trace norm regularization technique f…
We introduce a pruning algorithm that provably sparsifies the parameters of a trained model in a way that approximately preserves the model's predictive accuracy. Our algorithm uses a small batch of input points to construct a data-informed importance sampling distribution over the network's parameters, and adaptively …
Analyzes high-dimensional SGD dynamics using DMFT.
Mini-batch stochastic gradient descent (SGD) and variants thereof approximate the objective function's gradient with a small number of training examples, aka the batch size. Small batch sizes require little computation for each model update but can yield high-variance gradient estimates, which poses some challenges for…
Study on how optimal representations emerge during deep learning training, focusing on the role of implicit regularization.
Paper proposes an efficient method for bounding box annotation in object detection.
SNGM improves large-batch training accuracy.
Noise enhancement improves generalization in training.
BEMA reduces bias in EMA, leading to faster convergence and better performance.
Large batch training with DP-SGD reduces model performance due to implicit bias.
This paper studies the inference problem in quantile regression (QR) for a large sample size but under a limited memory constraint, where the memory can only store a small batch of data of size . A natural method is the naïve divide-and-conquer approach, which splits data into batches of size , computes the l…
Study predicts SGD test loss for structured features.
DReg boosts large-batch SGD's generalization and convergence.
Modern machine learning models are typically trained using Stochastic Gradient Descent (SGD) on massively parallel computing resources such as GPUs. Increasing mini-batch size is a simple and direct way to utilize the parallel computing capacity. For small batch an increase in batch size results in the proportional red…
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly und…
Integrates fairness guarantees into deep learning models.
sGPO optimizes RLVR training by balancing inference FLOPs and training efficiency.
It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding…
A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend them…
Paper resolves bias in ALFT training using generalized alignment games.
Many image processing tasks involve image-to-image mapping, which can be addressed well by fully convolutional networks (FCN) without any heavy preprocessing. Although empirically designing and training FCNs can achieve satisfactory results, reasons for the improvement in performance are slightly ambiguous. Our study i…
Improved loss scaling for stochastic momentum algorithms in high dimensions.
Empirical study on SGD hyperparameters and adversarial robustness.
Inference for GP models with non-Gaussian noises is computationally expensive when dealing with large datasets. Many recent inference methods approximate the posterior distribution with a simpler distribution defined on a small number of inducing points. The inference is accurate only when data points have strong corre…
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadratic models of the objective function. All of this appears to call for a full batch approach, but sinc…
Gradient descent fails to learn simple neural networks efficiently.
STAD adapts models to evolving time-based data shifts.
Batch-splitting (data-parallelism) is the dominant distributed Deep Neural Network (DNN) training strategy, due to its universal applicability and its amenability to Single-Program-Multiple-Data (SPMD) programming. However, batch-splitting suffers from problems including the inability to train very large models (due to…
Federated Learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning however comes at the cost of a significant communication overhead du…
ABS dynamically adjusts batch size based on policy stability, improving RL performance.
New method shows how order of gradient updates impacts stability and convergence in deep learning.
In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It bec…
Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded de…
SGD noise has no bias advantage in online learning, contrary to offline learning.
A robust approach compensates for small-data tasks in mixed linear regression.