Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
problem Understanding the implicit bias of batch normalization in linear models and neural networks.
method Analyzing gradient descent convergence on linear models and two-layer CNNs with batch normalization.
result Gradient descent with batch normalization in linear models converges to a uniform margin classifier with an exponential convergence rate.
Batch uniformization improves anomaly detection in sound data.
problem Anomaly scores for rare and frequent normal sounds are not uniform.
method Propose batch uniformization to minimize anomaly scores by weighting samples based on their density.
result Improves performance of unsupervised anomaly detection in sound data.
Paper optimizes TSK fuzzy systems for large datasets with MBGD and novel regularization.
problem Optimizing TSK fuzzy systems for large datasets with high dimensionality.
method Proposes MBGD with UR and BN for TSK fuzzy classifiers.
result UR and BN improve classification performance on various UCI datasets.
New research shows many batch selection methods for training work just as well as full batch training.
problem Finding optimal batch selection methods for training.
method Analysis of mini-batch Gradient Descent (GD) and Stochastic GD (SGD) with various batch selection rules.
result All mini-batch schedules, including deterministic ones, generalize optimally for smooth Lipschitz-convex/nonconvex/strongly-convex loss functions.
Walraswap solves batch auction pricing by finding optimal AMM swaps.
problem Executing all trade orders with optimal automated market makers (AMMs).
method Uses Brouwer's fixed-point theorem to find equilibrium prices.
result A solution to batch auction pricing problems in blockchain.
Paper proposes a new method for efficient hyperparameter optimization.
problem Challenging task of optimizing hyperparameters in machine learning.
method Sequential Uniform Design (SeqUD) strategy for adaptive and efficient exploration of hyperparameter space.
result The proposed SeqUD strategy outperforms existing methods in hyperparameter optimization.
New method improves generalization in deep learning models.
problem Improving generalization in overparameterized deep neural networks.
method Stochastic Gauss-Newton method with Levenberg-Marquardt damping and mini-batch sampling.
result Established finite-time convergence and non-asymptotic generalization bounds.
We study a mini-batch diversification scheme for stochastic gradient descent (SGD). While classical SGD relies on uniformly sampling data points to form a mini-batch, we propose a non-uniform sampling scheme based on the Determinantal Point Process (DPP). The DPP relies on a similarity measure between data points and g…
Optimizes query routing to LLMs under cost and resource constraints.
problem Non-uniform or adversarial batching in per-query routing methods leads to cost inefficiency.
method Batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch.
result Robust routing framework improves accuracy by 1-14% over non-robust methods.
We propose a general yet simple theorem describing the convergence of SGD under the arbitrary sampling paradigm. Our theorem describes the convergence of an infinite array of variants of SGD, each of which is associated with a specific probability law governing the data selection rule used to form mini-batches. This is…
GRAIN: Group Aggregation via Min-Norm Objective
problem Learning instability in large models
method Replacing mean aggregation with min-norm convex combination
result Improves performance and reduces variance
Online learning of linear operators between infinite-dimensional spaces is possible but with limitations.
problem Learning linear operators between infinite-dimensional Hilbert spaces in an online setting.
method Online learning approach for linear operators with bounded p-Schatten norm, proving impossibility for operator norm. result Separation between online learnability and uniform convergence for bounded linear operators.
As an efficient and scalable graph neural network, GraphSAGE has enabled an inductive capability for inferring unseen nodes or graphs by aggregating subsampled local neighborhoods and by learning in a mini-batch gradient descent fashion. The neighborhood sampling used in GraphSAGE is effective in order to improve compu…
We improve GANs by enforcing reproducibility and using non-uniform sampling.
problem Overrepresentation of certain samples in GANs' marginal log-likelihood.
method Enforce reproducibility through matching empirical distribution to prior, use non-uniform sampling for mini-batch selection.
result Improved quality and variety in generated samples, validated on CIFAR10, Fashion MNIST, and CelebA.
Paper presents a low-cost algorithm for bipartite ranking with improved sample size requirements.
problem Bipartite ranking's quadratic dependence on sample size makes it computationally expensive.
method Uses a novel uniform risk bound based on matrix and vector concentration inequalities to achieve low cost and competitive performance.
result Shows that the sample size required for competitive performance is not quadratic, improving efficiency.
Cyclic Data Parallelism reduces memory usage and balances gradient communications.
problem Training large deep learning models requires efficient parallelism to scale.
method Cyclic Data Parallelism shifts micro-batches from simultaneous to sequential execution, balancing memory and gradient communications.
result Cyclic Data Parallelism reduces total memory usage and balances gradient communications.
In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It bec…
Improved sampling accuracy in SG-MCMC methods via non-uniform gradient subsampling.
problem Computational inefficiency and sampling error in stochastic gradient MCMC methods.
method Proposes a non-uniform subsampling scheme to reduce sampling error in EWSG, a variant of SG-MCMC.
result EWSG reduces sampling error compared to uniform subsampling, improving accuracy without sacrificing convergence speed.
Training neural networks is traditionally done by providing a sequence of random mini-batches sampled uniformly from the entire training data. In this work, we analyze the effect of curriculum learning, which involves the non-uniform sampling of mini-batches, on the training of deep networks, and specifically CNNs trai…
New stability bounds for SGD on nonsmooth convex losses.
problem Understanding stability of SGD on nonsmooth convex losses.
method Sharp upper and lower bounds for SGD and full-batch GD on nonsmooth convex losses.
result SGD can be less stable but still useful for generalization bounds.
Modern applications in sensitive domains such as biometrics and medicine frequently require the use of non-decomposable loss functions such as precision@k, F-measure etc. Compared to point loss functions such as hinge-loss, these offer much more fine grained control over prediction, but at the same time present novel c…
We consider the problem of learning a non-negative linear classifier with a 1-norm of at most k, and a fixed threshold, under the hinge-loss. This problem generalizes the problem of learning a k-monotone disjunction. We prove that we can learn efficiently in this setting, at a rate which is linear in both k and…
New methods improve convergence in non-convex non-smooth learning problems.
problem Sparse learning from high-dimensional data with non-convex, non-smooth regularizers.
method Stochastic proximal gradient methods with arbitrary sampling.
result Independent sampling improves performance over uniform sampling.
We investigate 1) the rate at which refined properties of the empirical risk---in particular, gradients---converge to their population counterparts in standard non-convex learning tasks, and 2) the consequences of this convergence for optimization. Our analysis follows the tradition of norm-based capacity control. We p…
Study learns random hypergraphs with queries, improving on previous results.
problem Learn random hypergraphs with non-adaptive queries.
method Equivalence to group testing, using Erdős-Rényi model for graphs.
result Generalization to random k-uniform hypergraphs. New analysis shows SGD with noise doesn't leak more privacy with more iterations.
problem Privacy loss in noisy SGD with more iterations.
method Privacy Amplification by Iteration and Sampled Gaussian Mechanism.
result Privacy loss remains constant after a burn-in period, not increasing with more iterations.
While stochastic gradient descent (SGD) is one of the major workhorses in machine learning, the learning properties of many practically used variants are poorly understood. In this paper, we consider least squares learning in a nonparametric setting and contribute to filling this gap by focusing on the effect and inter…
We consider a generic convex optimization problem associated with regularized empirical risk minimization of linear predictors. The problem structure allows us to reformulate it as a convex-concave saddle point problem. We propose a stochastic primal-dual coordinate (SPDC) method, which alternates between maximizing ov…
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
problem Batch Normalization's dependency on mini-batch elements can degrade performance for small batches.
method Filter Response Normalization (FRN) operates independently on each activation channel of each batch element.
result FRN layer outperforms BN and other alternatives in various settings for all batch sizes.
We analyze how batch learning impacts bandit problems.
problem Impact of batch learning in stochastic bandits.
method Policy-agnostic regret analysis, upper and lower bounds demonstration.
result The impact of batch learning can be measured in terms of online behavior.
BatchGFN uses generative flow networks for efficient batch active learning.
problem Efficiently selecting informative batches for active learning.
method Generative flow networks to sample batches proportional to a batch reward.
result Constructs highly informative batches with a single forward pass per point.
Adaptive batch sizes improve active learning efficiency and flexibility.
problem Fixed batch sizes in active learning are inefficient due to dynamic cost-speed trade-offs.
method Probabilistic Numerics framework that adaptively changes batch sizes based on integration error and precision objectives.
result Significant enhancement in learning efficiency and flexibility across various applications.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.
problem Batched best arm identification (BBAI) problem, aiming to minimize policy switches and resource usage.
method Proposed Opt-BBAI algorithm, achieving near-optimal sample and batch complexity in non-asymptotic settings.
result First algorithm to achieve near-optimal sample and batch complexity in non-asymptotic settings.
A new method designs batches for Bayesian optimization more efficiently.
problem Efficiently designing batches for Bayesian optimization to reduce total time.
method Minimal Terminal Variance (MTV) acquisition function, optimizing I-optimality criterion.
result MTV designs batches more efficiently than other methods, as shown by numerical experiments.
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
Four improvements to Batch Normalization improve deep learning performance.
problem Improving Batch Normalization for better deep learning performance.
method Proposed improvements include reasoning about current examples, Ghost Batch Normalization, weight decay regularization, and a new normalization algorithm for small batch sizes.
result Performance gains across all batch sizes with no additional computation during training.
Riemannian stochastic gradient descent converges faster with increasing batch size.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Theoretical analysis and numerical investigation of increasing batch size effects.
result Riemannian stochastic gradient descent converges faster with increasing batch size.
Polynomial-time algorithm for list-decodable linear regression with batches.
problem Efficiently decoding linear regression with a fraction of adversarial data.
method Polynomial time algorithm using batches of i.i.d. samples.
result Returns a list of size O(1/α^2) with one item close to true parameter.
New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.
problem Understanding the stability and convergence of mini-batch SGD.
method Analyzing the mini-batch Hessian and its directional curvature.
result Mini-batch SGD operates in a different stability regime (Edge of Stochastic Stability) compared to full-batch GD.
Batch normalisation doesn't affect variational inference but fails for larger batch sizes.
problem Failure of Monte Carlo Batch Normalisation (MCBN) for capturing epistemic uncertainty in larger batch sizes.
method Investigated MCBN as an approximate inference technique for Bayesian neural networks, showing its limitations and providing insights for improvement.
result For larger batch sizes, MCBN fails to capture epistemic uncertainty, requiring the batch size to be a variational parameter.
A new scaling law predicts optimal batch size for training models.
problem Finding the optimal batch size for training models efficiently.
method Proposed a three-term scaling law that considers model size, training data, training steps, and batch size.
result The three-term law accurately recovers the optimal batch size and can be robustly fit with fewer training runs.
E4 algorithm optimizes batched linear bandits with minimal regret and batches.
problem Optimizing batched linear bandits for minimal regret.
method Explore-Estimate-Eliminate-Exploit framework with optimal exploration rate.
result Achieves minimax and asymptotic optimality in regret and batch complexity.
ABS dynamically adjusts batch size based on policy stability, improving RL performance.
problem Diminishing returns with large batch sizes in RL due to non-stationary data.
method Adaptive Batch Scaling (ABS) with Behavioral Divergence metric.
result Larger batch sizes can improve RL performance, contrary to conventional wisdom.
SNGM improves large-batch training accuracy.
problem Improving generalization in large-batch training.
method Stochastic Normalized Gradient Descent with Momentum.
result SNGM achieves better test accuracy than MSGD and other large-batch methods.
Paper proposes MABN to stabilize BN in small batch sizes.
problem Weakness of BN in small batch sizes limits its use in tasks with small batch sizes.
method Identifies and addresses two extra batch statistics in BN's backward propagation.
result MABN completely restores BN's performance in small batch sizes without additional nonlinear operations.
Batched Neural Bandits reduces policy updates in sequential decision-making.
problem Sequential decision-making with batched policy changes.
method BatchNeuralUCB algorithm combining neural networks and optimism.
result Achieves similar regret as fully sequential version with fewer policy updates.