A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Combines multi-fidelity and asynchronous batch methods for faster experimental design.
A new method designs batches for Bayesian optimization more efficiently.
We analyze how batch learning impacts bandit problems.
Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of m…
Improved batched SH algorithm maintains original performance.
This paper tackles batch Bayesian optimal experimental design by using Wasserstein gradient flows.
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and…
We propose a new batch mode active learning algorithm designed for neural networks and large query batch sizes. The method, Discriminative Active Learning (DAL), poses active learning as a binary classification task, attempting to choose examples to label in such a way as to make the labeled set and the unlabeled pool …
Adaptive batch sizes improve active learning efficiency and flexibility.
This work uses RL to optimize batch experiments in SDOE.
A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested …
Mini-batch gradient descent based methods are the de facto algorithms for training neural network architectures today. We introduce a mini-batch selection strategy based on submodular function maximization. Our novel submodular formulation captures the informativeness of each sample and diversity of the whole subset. W…
This paper finds ReLU restores symmetry in SCL under class imbalances.
To deal with very large datasets a mini-batch version of the Monte Carlo Markov Chain Stochastic Approximation Expectation-Maximization algorithm for general latent variable models is proposed. For exponential models the algorithm is shown to be convergent under classicalconditions as the number of iterations increases…
We propose the online machine learning for big data analysis with heterogeneity. We performed an experiment to compare the accuracy of each iteration between batch one and online one. It is possible to converge quickly with the same accuracy as the batch one.
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.
Adaptive SGD learns optimal batch size for strong convex functions.
Adaptive batching improves Gaussian process surrogates for noisy level set estimation.
iMondrian forest combines isolation forest and Mondrian forest for better anomaly detection.
Develops a flexible batched experimentation framework for limited adaptivity.
Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, w…
Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-siz…
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, thr…
E algorithm optimizes batched linear bandits with minimal regret and batches.
DPP-BBO diversifies batched Bayesian optimization using DPPs.
TaskNorm improves meta-learning performance by rethinking batch normalization.
This paper explores the limits of large batch sizes in deep learning.
Proposes m-POT to improve m-OT's misspecified mappings issue.
Novel algorithm reduces feature inclusion in online decision-making.
Adaptive batch sizes improve local gradient methods in distributed training.
New algorithms for batched dueling bandits with improved regret bounds.
In this work we compare different batch construction methods for mini-batch training of recurrent neural networks. While popular implementations like TensorFlow and MXNet suggest a bucketing approach to improve the parallelization capabilities of the recurrent training process, we propose a simple ordering strategy tha…
Optimizes query routing to LLMs under cost and resource constraints.
Mini-batch stochastic gradient descent (SGD) and variants thereof approximate the objective function's gradient with a small number of training examples, aka the batch size. Small batch sizes require little computation for each model update but can yield high-variance gradient estimates, which poses some challenges for…
A new algorithm learns policies from batch data in hierarchical RL.
Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. However, the optimizer converges slowly at early epochs and there is a gap between large-batch deep learning optimization heuristics and theoretical underpinnings. In this paper, we propose a no…
New analysis reveals batch size effects on stochastic conditional gradient methods.
There has recently been a surge in research in batch Deep Reinforcement Learning (DRL), which aims for learning a high-performing policy from a given dataset without additional interactions with the environment. We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and per…
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
Algorithm achieves comparable performance to fully dynamic data with only a few batches.
We present K-Means Batch Bayesian Optimization (KMBBO), a novel batch sampling algorithm for Bayesian Optimization (BO). KMBBO uses unsupervised learning to efficiently estimate peaks of the model acquisition function. We show in empirical experiments that our method outperforms the current state-of-the-art batch alloc…
Study batch learning in linear bandits with context, achieving near-optimal performance.
Adam's generalization performance is improved by batch size and weight decay in neural networks.
A new method for mini-batch optimal transport improves scalability and accuracy.
Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.
The popularity of Bayesian optimization methods for efficient exploration of parameter spaces has lead to a series of papers applying Gaussian processes as surrogates in the optimization of functions. However, most proposed approaches only allow the exploration of the parameter space to occur sequentially. Often, it is…
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find th…
In this work we develop a new algorithm for regularized empirical risk minimization. Our method extends recent techniques of Shalev-Shwartz [02/2015], which enable a dual-free analysis of SDCA, to arbitrary mini-batching schemes. Moreover, our method is able to better utilize the information in the data defining the ER…