The paper derives a new theorem for predicting batches of data.
problem Finding lower bounds on minimal batch regret.
method Derives a conditional version of the regret-capacity theorem.
result Reveals a connection between conditional Rényi divergence and conditional Sibson's mutual information.
New method evaluates LLMs fairness in universal prediction.
problem Evaluating fairness of large language models in universal prediction.
method Introducing batch regret as a modification of average regret for LLMs.
result Asymptotical value of batch regret for add-constant predictors on memoryless and first-order Markov sources.
Exact distribution of split conformal prediction coverage found.
problem Determining the reliability of prediction sets in batch mode.
method Analysis of exchangeable data to find universal distribution of empirical coverage.
result Exact distribution of empirical coverage is universal and determined by nominal miscoverage level and calibration sample size.
Universal supervised learning is considered from an information theoretic point of view following the universal prediction approach, see Merhav and Feder (1998). We consider the standard supervised "batch" learning where prediction is done on a test sample once the entire training data is observed, and the individual s…
The goal of this paper is to characterize function distributions that deep learning can or cannot learn in poly-time. A universality result is proved for SGD-based deep learning and a non-universality result is proved for GD-based deep learning; this also gives a separation between SGD-based deep learning and statistic…
We analyze the learning properties of the stochastic gradient method when multiple passes over the data and mini-batches are allowed. We study how regularization properties are controlled by the step-size, the number of passes and the mini-batch size. In particular, we consider the square loss and show that for a unive…
A new method for batch prediction sets in classification problems.
problem Constructing reliable prediction sets for multiple unlabeled examples.
method Proposes a uniformly more powerful approach to batch prediction sets using specific combinations of conformal p-values.
result The proposed method provides narrower prediction sets compared to the Bonferroni correction.
New acquisition function improves batch Bayesian active learning.
problem BatchBALD conflates epistemic and aleatoric uncertainty, leading to suboptimal performance.
method Focus on predictive probabilities to separate epistemic uncertainty, leading to better performance and faster evaluation.
result The new acquisition function performs better and allows for larger batches.
DPP-BBO diversifies batched Bayesian optimization using DPPs.
problem Efficiently proposing diverse and informative batches in batched Bayesian optimization.
method Introducing DPP-Batch Bayesian Optimization (DPP-BBO) with DPP-Thompson Sampling (DPP-TS).
result Novel Bayesian simple regret bounds for DPP-TS show improved performance over classical methods.
New study shows non-adaptive trials can be outperformed by adaptive designs in treatment selection.
problem Determining the best allocation of resources in clinical trials.
method Analysis of batched arm elimination designs and comparison with completely randomized trials.
result Simple adaptive designs universally and strictly dominate non-adaptive completely randomized trials for at least three treatment arms.
Batch Active Learning uses derivative information for Gaussian Process regression.
problem Efficiently selecting data batches in Gaussian Process regression models.
method Proposes using the predictive covariance matrix to select data batches, exploiting full correlation.
result Demonstrates the effectiveness of incorporating derivative information across diverse applications.
New analysis reveals batch size effects on stochastic conditional gradient methods.
problem Understanding the role of batch size in stochastic conditional gradient methods.
method Deriving a new analysis focusing on momentum-based stochastic conditional gradient algorithms (e.g., Scion).
result Increasing batch size initially improves optimization accuracy but can degrade performance beyond a critical threshold.
Improved stock return prediction model handles noise and non-stationarity.
problem Predicting stock returns with robustness to noise and non-stationarity.
method Extended AROW algorithm to handle synchronous mini-batch updates and applied it to stock return prediction.
result The new model outperforms classical approaches in backtesting on S\&P500 stocks.
Proposes reducing random error in stochastic optimization by variance regularization.
problem Random error accumulation in stochastic optimization algorithms.
method Regularizes learning-rate based on mini-batch variances.
result Speeds up convergence and stabilizes stochastic optimization.
We develop parallel predictive entropy search (PPES), a novel algorithm for Bayesian optimization of expensive black-box objective functions. At each iteration, PPES aims to select a batch of points which will maximize the information gain about the global maximizer of the objective. Well known strategies exist for sug…
The accuracy of deep neural networks is significantly affected by how well mini-batches are constructed during the training step. In this paper, we propose a novel adaptive batch selection algorithm called Recency Bias that exploits the uncertain samples predicted inconsistently in recent iterations. The historical lab…
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, thr…
New method for scalable set encoding with unbiased gradient approximation.
problem Limited expressive power and large set training issues in set functions.
method Universally MBC (UMBC) class of set functions and efficient MBC training algorithm.
result Unbiased approximation of full set gradient with constant memory overhead.
A new algorithm PBNN improves scalability of Bayesian Neural Networks.
problem Scalability issues in Neural Network posterior sampling with large datasets.
method Penalty Bayesian Neural Networks (PBNN) using subsampled batch data.
result PBNN achieves good predictive performance with small mini-batch sizes.
Develops a flexible batched experimentation framework for limited adaptivity.
problem Challenges of continual reallocation in bandit algorithms with delayed feedback.
method Computational framework leveraging Gaussian sequential experiment and dynamic programming.
result Improves statistical power over standard methods, even compared to Bayesian bandit algorithms.
A new scaling law predicts optimal batch size for training models.
problem Finding the optimal batch size for training models efficiently.
method Proposed a three-term scaling law that considers model size, training data, training steps, and batch size.
result The three-term law accurately recovers the optimal batch size and can be robustly fit with fewer training runs.
Prediction-time batch normalization improves model robustness under covariate shift.
problem Dealing with covariate shift in deep learning models.
method Prediction-time batch normalization, a simple but effective method.
result Significantly improves model accuracy and calibration under covariate shift.
New method predicts and optimizes matrix recovery from noisy measurements.
problem Recovering rank-1 matrices from Gaussian measurements with noise.
method Stochastic prox-linear iterative algorithm with trajectory predictions.
result The method converges linearly with accurate predictions of error.
Ensembles of randomized decision trees, usually referred to as random forests, are widely used for classification and regression tasks in machine learning and statistics. Random forests achieve competitive predictive performance and are computationally efficient to train and test, making them excellent candidates for r…
We design a new algorithm for batch active learning with deep neural network models. Our algorithm, Batch Active learning by Diverse Gradient Embeddings (BADGE), samples groups of points that are disparate and high-magnitude when represented in a hallucinated gradient space, a strategy designed to incorporate both pred…
VQ-GNN scales GNNs to large graphs using vector quantization.
problem Scaling GNNs to large graphs with stable performance and speed.
method VQ-GNN uses vector quantization to preserve all messages passed to a mini-batch of nodes, avoiding the 'neighbor explosion' problem.
result VQ-GNN achieves competitive performance on large-graph node classification and link prediction benchmarks.
The paper studies batch decompositions of random datasets with probabilistic similarity constraints.
problem Understanding how to optimally split large datasets into batches for better model learning.
method Assumes independent data points from a space, defines similarity, and uses probabilistic and martingale methods to find bounds on batch sizes.
result Demonstrates an inherent tradeoff between relaxing similarity constraints and batch size, and provides bounds for maximum similarity subsets.
Bayesian optimisation is a popular, surrogate model-based approach for optimising expensive black-box functions. Given a surrogate model, the next location to expensively evaluate is chosen via maximisation of a cheap-to-query acquisition function. We present an ε-greedy procedure for Bayesian optimisation in batch s…
The study improves compound selection in in silico screening by focusing on model's ability to predict desirable outcomes.
problem Improving compound selection in in silico screening to reduce errors and enhance generalization.
method Extending learning theory, the study analyzes the impact of selection policies on generalization and proposes a method to mitigate challenges.
result Generalization can be enhanced by considering a model's ability to predict the fraction of desired outcomes in a batch.
Proposes a new batch selection method for multi-label classification.
problem Improving the accuracy of deep neural networks in multi-label classification tasks.
method Adapts uncertainty measures to multi-label data, considering label correlations and dynamic uncertainty.
result Improves performance and accelerates convergence of multi-label deep learning models.
TS-RSR improves batch Bayesian Optimization by minimizing redundancy and focusing on high uncertainty points.
problem Efficient batch selection in Bayesian Optimization to reduce redundancy and improve performance.
method TS-RSR uses Thompson Sampling to minimize a regret to uncertainty ratio for batch selection.
result TS-RSR outperforms state-of-the-art batch BO algorithms on various test functions.
Anytime-valid confirmation of label-shift corrections
problem Small-batch scientific deployments with scarce labeled outcomes
method Conditional e-value and martingale-based rule
result Nonnegative martingale and anytime-valid confirmation rule
Optimizes query routing to LLMs under cost and resource constraints.
problem Non-uniform or adversarial batching in per-query routing methods leads to cost inefficiency.
method Batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch.
result Robust routing framework improves accuracy by 1-14% over non-robust methods.
Universal online optimization for dynamic environments using uniclass prediction.
problem Online optimization in changing environments with dynamic regret.
method Reduces dynamic online optimization to uniclass prediction problem, allowing control over dynamic regret bounds.
result First paper with state-of-the-art dynamic regret guarantees for general convex cost functions.
BYOL learns useful representations without batch statistics.
problem BYOL's reliance on batch statistics for representation learning.
method Training BYOL without batch normalization and using alternative normalization schemes.
result BYOL performance comparable to vanilla BYOL without batch normalization.
Paper evaluates different models for predicting credit default swap volatility.
problem Predicting the Implied Volatility of credit default swaps.
method SVM, Gradient Boosting, and Attention-GRU Hybrid model.
result Identifies strengths in classical and SOTA machine learning methods.
In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to …
A new method improves active learning for large batch sizes.
problem Challenges in scaling Bayesian active learning to large batch sizes.
method Derives Partial Batch Label Sampling (ParBaLS) for EPIG algorithm.
result ParBaLS EPIG outperforms top-B selection and BatchBALD. Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models. In this paper we develop a new analytic framework to analyze noise-averaged properties of mini-batch SGD for linear models at constant learning rates, momenta and sizes of batches. Our key idea is to consider the dynamics of t…
Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.
problem Understanding how Adam's implicit bias is affected by mini-batch size and momentum parameters.
method Theoretical framework to analyze mini-batch noise's impact on Adam's memory and bias.
result The magnitude of anti-regularization by memory depends on batch size and momentum hyperparameters.
Extends batch active learning to non-differentiable models.
problem Efficiently training machine learning models on large, initially unlabelled datasets.
method Black-box batch active learning for regression tasks that relies solely on model predictions.
result Achieves strong performance on regression datasets compared to white-box approaches for deep learning models.
Batch-splitting (data-parallelism) is the dominant distributed Deep Neural Network (DNN) training strategy, due to its universal applicability and its amenability to Single-Program-Multiple-Data (SPMD) programming. However, batch-splitting suffers from problems including the inability to train very large models (due to…
Modeling a sequence of interactions between users and items (e.g., products, posts, or courses) is crucial in domains such as e-commerce, social networking, and education to predict future interactions. Representation learning presents an attractive solution to model the dynamic evolution of user and item properties, w…
Improved deep probabilistic time series forecasting by learning error autocorrelation.
problem Simplification of time-independent error process and lack of serial correlation in existing models.
method Proposes a training method that incorporates error autocorrelation to enhance probabilistic forecasting accuracy.
result Improves predictive accuracy and uncertainty quantification across multiple datasets.
We demonstrate the existence of universal adversarial perturbations, which can fool a family of audio classification architectures, for both targeted and untargeted attack scenarios. We propose two methods for finding such perturbations. The first method is based on an iterative, greedy approach that is well-known in c…
New framework optimizes deep learning training by deferring large batch sizes to late stages.
problem Optimizing batch size scheduling for deep learning training efficiency.
method Introduced the functional scaling law (FSL) framework to analyze and optimize batch size scheduling.
result Large batch sizes can be deferred to late training stages without sacrificing performance.
We introduce a new, high-throughput, synchronous, distributed, data-parallel, stochastic-gradient-descent learning algorithm. This algorithm uses amortized inference in a compute-cluster-specific, deep, generative, dynamical model to perform joint posterior predictive inference of the mini-batch gradient computation ti…
We present a unified framework for Batch Online Learning (OL) for Click Prediction in Search Advertisement. Machine Learning models once deployed, show non-trivial accuracy and calibration degradation over time due to model staleness. It is therefore necessary to regularly update models, and do so automatically. This p…