Paper develops a method to construct confidence regions for model parameters using batch means method.
problem Constructing confidence regions for model parameters in stochastic gradient descent.
method Batch means method to cancel out covariance matrix, using Polyak-Ruppert averaging.
result Established process-level functional central limit theorem for stochastic gradient descent estimators.
A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested …
We present K-Means Batch Bayesian Optimization (KMBBO), a novel batch sampling algorithm for Bayesian Optimization (BO). KMBBO uses unsupervised learning to efficiently estimate peaks of the model acquisition function. We show in empirical experiments that our method outperforms the current state-of-the-art batch alloc…
Stein shrinkage improves BN robustness against adversarial attacks.
problem Improving BN robustness against adversarial attacks.
method Applying Stein shrinkage to BN mean and variance estimates.
result Stein shrinkage outperforms vanilla BN in adversarial settings.
Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (A…
Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by d…
Batch normalisation doesn't affect variational inference but fails for larger batch sizes.
problem Failure of Monte Carlo Batch Normalisation (MCBN) for capturing epistemic uncertainty in larger batch sizes.
method Investigated MCBN as an approximate inference technique for Bayesian neural networks, showing its limitations and providing insights for improvement.
result For larger batch sizes, MCBN fails to capture epistemic uncertainty, requiring the batch size to be a variational parameter.
Improves deep learning training by matching mini-batch distributions.
problem Overfitting and noise in mini-batch training.
method ITDM, which matches the moments of mini-batch distributions to reduce overfitting.
result ITDM reduces overfitting and improves DNN training.
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute…
Use of an autoencoder (AE) as a normal model is a state-of-the-art technique for unsupervised-anomaly detection in sounds (ADS). The AE is trained to minimize the sample mean of the anomaly score of normal sounds in a mini-batch. One problem with this approach is that the anomaly score of rare-normal sounds becomes hig…
New algorithms minimize MMD to approximate probability measures efficiently.
problem Approximating probability measures by representative point sets.
method Sequential greedy minimization of maximum mean discrepancy (MMD) over candidate sets, with mini-batch variants.
result Consistency of proposed algorithms and mini-batch variants established.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
Adaptive batching improves Gaussian process surrogates for noisy level set estimation.
problem Learning the level set of noisy simulator responses.
method Developed four novel adaptive batching schemes for Gaussian process metamodels.
result Adaptive batching brings significant computational speed-ups with minimal loss of modeling fidelity.
TS-RSR improves batch Bayesian Optimization by minimizing redundancy and focusing on high uncertainty points.
problem Efficient batch selection in Bayesian Optimization to reduce redundancy and improve performance.
method TS-RSR uses Thompson Sampling to minimize a regret to uncertainty ratio for batch selection.
result TS-RSR outperforms state-of-the-art batch BO algorithms on various test functions.
This paper tackles batch Bayesian optimal experimental design by using Wasserstein gradient flows.
problem The challenge of optimising high-dimensional, strongly non-convex expected information gain in batch settings.
method Probabilistic lifting to the space of probability measures, entropic regularisation, Wasserstein gradient flow, and particle-based algorithms.
result The proposed approach can be used directly as a randomised batch-design policy or as a computational relaxation.
Two-layer networks learn faster with batch reuse, overcoming information and leap exponents.
problem Limitations of gradient flow and single-pass GD in learning multi-index target functions.
method Multi-pass gradient descent that reuses batches, analyzed using Dynamical Mean-Field Theory.
result Two-time-step overlap with target subspace for non-staircase functions, overcoming information and leap exponents.
Differentiates Fréchet mean for hyperbolic space applications.
problem Difficulty in applying Fréchet mean due to lack of closed-form derivative.
method Developed differentiation method and explicit gradient expressions for hyperbolic space.
result Fully integrated Fréchet mean into hyperbolic neural network pipeline.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …
New algorithm tackles batched stochastic linear bandits with 1-bit communication constraints.
problem Stochastic linear bandits with 1-bit communication constraints.
method Phased-elimination algorithms based on G-optimal designs and 1-bit mean estimation.
result Achieves near-optimal regret bounds for broad scaling regimes.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- \textit{Internal Covariate Shift}-- the current solution has certain drawbacks. For instance, BN depends on batch statistics for layerwise input normalization during training which makes the esti…
New Convolutional Unit improves Batch Whitening performance.
problem Improving the efficiency and effectiveness of Batch Whitening.
method Proposes a new Convolutional Unit that aligns with Batch Whitening theory and empirically analyzes the original Convolutional Unit.
result Significantly improved performance on multiple image classification datasets.
Sources of variability in experimentally derived data include measurement error in addition to the physical phenomena of interest. This measurement error is a combination of systematic components, originating from the measuring instrument, and random measurement errors. Several novel biological technologies, such as ma…
Optimal algorithm for minimizing regret in heavy-tailed bandits.
problem Minimizing regret in stochastic multi-armed bandits with heavy-tailed distributions.
method Proposes an optimal algorithm under the assumption of uniformly bounded moments of order (1+ε).
result Matches the lower bound exactly in the first-order term and provides a finite-time bound on its regret.
QC-ST and CoCo methods correct batch effects in metabolomics data.
problem Batch effects in metabolomics data obscure biological variations.
method QC-ST for simultaneous detection of QC samples' mean vectors and covariance matrices, CoCo for covariance correction.
result QC-ST and CoCo improve batch effect correction in metabolomics datasets.
Improved TD learning reduces batch sampling error.
problem TD learning can be inaccurate in batch settings due to action weighting.
method Introduced PSEC-TD(0) to correct action weighting using importance sampling.
result PSEC-TD(0) produces more accurate value function estimates.
STEEL tackles batch RL with singularity, improving policy optimization.
problem Existing RL methods assume absolutely continuous data, but STEEL handles non-overlapping regions.
method Proposes STEEL algorithm using maximum mean discrepancy and distributionally robust optimization.
result First finite-sample regret guarantee for batch RL with singularity.
Improved backpropagation with consequentialism weight updates for neural networks.
problem Improving backpropagation for neural networks, especially with mini-batch training.
method Introducing consequentialism weight updates derived from NLMS for multi-layer neural networks.
result The proposed method outperforms traditional BP and mini-batch training.
We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient i…
The paper provides concentration inequalities for Markov chain variance estimators.
problem Estimating the variance of Markov chains with concentration properties.
method Martingale decomposition method for uniformly geometrically ergodic Markov chains.
result Explicit control of the p-th moment of the OBM estimator difference and dependence on p and mixing time.
Study SGD dynamics in high-dimensional models, revealing consistent behavior across different batch sizes and learning rates.
problem Understanding SGD dynamics in high-dimensional multi-index models.
method Asymptotic analysis of SGD, developing mean-field equations and Gaussian diffusion approximations.
result Consistent SGD dynamics across different batch sizes and learning rates, distinct from gradient flow and online SGD.
GRAIN: Group Aggregation via Min-Norm Objective
problem Learning instability in large models
method Replacing mean aggregation with min-norm convex combination
result Improves performance and reduces variance
New algorithms learn from untrusted batches with improved efficiency.
problem Learning from untrusted batches with adversarial responses.
method Sum-of-Squares hierarchy applied to robust mean estimation.
result Reduces sample complexity to polylogarithmic in n for most natural distributions. New technique trains deep neural networks without normalization or minibatch statistics.
problem Training deep neural networks at high learning rates without normalization.
method Channel-wise zero-mean initialization and gradient modification to maintain common mode rejection.
result Achieves higher accuracy compared to batch normalization and shows minibatches are unnecessary.
We study the problem of reducing the amount of labeled training data required to train supervised classification models. We approach it by leveraging Active Learning, through sequential selection of examples which benefit the model most. Selecting examples one by one is not practical for the amount of training examples…
A new algorithm for K-means clustering in evolving data streams.
problem Clustering of continuously arriving data in streaming scenarios with concept drift.
method Formal definition of Streaming K-means, surrogate error function, algorithm for minimizing surrogate error. result The surrogate error function effectively approximates the Streaming K-means error. This research investigates if deep neural networks can be trained without batch normalization.
problem Training deep neural networks efficiently without batch normalization.
method Detailed study of batch normalization, comparison with other methods, and adaptation of training process.
result It is possible to train deep neural networks effectively without batch normalization.
Introduce a variance-weighted batch distribution for diverse sampling in diffusion models.
problem Independent sampling in diffusion models.
method Introduce a variance-weighted batch distribution.
result Sampler with a transparent probabilistic target.
CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching
problem Inference of tabular foundation models with large training datasets
method CRUMB (Clustered Retrieval Using Minimised-MMD Batching)
result CRUMB outperforms state-of-the-art context selection strategies on the TabArena benchmark
We address the problem of estimating statistics of hidden units in a neural network using a method of analytic moment propagation. These statistics are useful for approximate whitening of the inputs in front of saturating non-linearities such as a sigmoid function. This is important for initialization of training and f…
Batch Normalization (BatchNorm) is an extremely useful component of modern neural network architectures, enabling optimization using higher learning rates and achieving faster convergence. In this paper, we use mean-field theory to analytically quantify the impact of BatchNorm on the geometry of the loss landscape for …
Study improves BN TTA under distribution shift using higher-order asymptotics.
problem Improving BN TTA for changing data distributions.
method Integrates Edgeworth expansion and saddlepoint approximation with one-step M-estimation.
result Derives optimal weighting parameter for minimized mean-squared error.
ε-shotgun optimizes expensive function evaluations in parallel.
problem Optimizing expensive black-box functions efficiently.
method ε-greedy batch Bayesian optimisation.
result ε-shotgun outperforms state-of-the-art batch methods.
This paper explains how batch normalization auto-tunes the regularization parameter based on data statistics.
problem Batch normalization accelerates deep learning training but the exact relationship to regularization is unclear.
method Theoretical analysis and empirical validation of batch normalization's role in auto-tuning the regularization parameter.
result Batch normalization auto-tunes the regularization parameter based on data statistics.
Recently, a lot of effort has been paid to the efficient computation of Kriging predictors when observations are assimilated sequentially. In particular, Kriging update formulae enabling significant computational savings were derived in Barnes and Watson (1992), Gao et al. (1996), and Emery (2009). Taking advantage of …
When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges. Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a colle…
Paper analyzes gradient descent with noisy data copies for linear regression, showing regularization and acceleration effects.
problem Improving generalization in machine learning through data augmentation with noise.
method Gradient descent with on-line noisy copies for linear regression analysis.
result Training with on-line noisy copies is equivalent to ridge regularization with a specific regularization parameter.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Study mini-batch SGD noise and its limits, proving complexity guarantees.
problem Analyzing the noise in mini-batch SGD and its impact on optimization.
method Examined the conditional covariance and diffusion limits of SGD under different sampling designs.
result Proved mean-square upper bounds and Fisher van Trees lower bounds for SGD, linking them to effective dimension and condition number.