New method improves GANs by training a mixed batch discriminator.
problem GANs struggle with mode collapse due to focusing on individual samples.
method Train a discriminator on mixed batches of true and fake samples.
result Significantly reduces mode collapse in GANs on various datasets.
SOBER optimizes and quadrates efficiently in parallel for diverse tasks.
problem Scalability of batch Bayesian optimization and quadrature for expensive functions.
method Reformulates batch selection as a quadrature problem, balancing exploitation and exploration.
result SOBER outperforms 11 baselines on 12 tasks.
A new method for statistical inference using SGD under φ-mixing data.
problem Valid statistical inference for time series data with general correlation.
method Proposes a mini-batch SGD estimator and associated mini-batch bootstrap procedure for φ-mixing data. result The proposed method constructs valid confidence intervals for φ-mixing data. Batchboost stabilizes training by mixing and pairing samples, improving accuracy.
problem Stabilizing training in machine learning, especially avoiding overfitting and underfitting.
method Batchboost pipeline with three stages: pairing, mixing, and feeding. Mixing uses mixup technique.
result Batchboost achieves 0.5-3% better accuracy than mixup on CIFAR-10 & Fashion-MNIST.
Joint sparsity offers powerful structural cues for feature selection, especially for variables that are expected to demonstrate a "grouped" behavior. Such behavior is commonly modeled via group-lasso, multitask lasso, and related methods where feature selection is effected via mixed-norms. Several mixed-norm based spar…
Trains large RNN model on 40GB of text in 4 hours.
problem Training large-scale language models efficiently.
method Mixed precision arithmetic, 32k batch size, 128 NVIDIA GPUs, learning rate schedule.
result Trains character-level LSTM over 40GB of Amazon Reviews in 4 hours.
SOBER framework optimizes Bayesian optimization tasks efficiently.
problem Challenges in parallel Bayesian optimization.
method Probabilistic Lifting with Kernel Quadrature.
result Versatile and flexible batch Bayesian optimization.
A new method optimises problems with both continuous and categorical inputs.
problem Optimising black-box problems with mixed continuous and categorical inputs.
method Continuous and Categorical Bayesian Optimisation (CoCaBO) combining multi-armed bandits and Bayesian optimisation.
result CoCaBO outperforms existing methods on synthetic and real-world tasks.
Our system trains ImageNet in 6.6 minutes using 2048 Tesla P40 GPUs.
problem Training large-scale deep neural networks efficiently and accurately.
method Mixed-precision training, extremely large mini-batch size optimization, and optimized all-reduce algorithms.
result Trains ResNet-50 to 75.8% top-1 test accuracy in 6.6 minutes.
i-Mix improves contrastive learning across domains without domain-specific augmentations.
problem Improving contrastive representation learning for unlabeled data across diverse domains.
method i-Mix treats contrastive learning as a non-parametric classifier problem, mixing data in input and virtual label spaces.
result i-Mix consistently improves representation quality across image, speech, and tabular data domains.
In this paper we address the following question: Can we approximately sample from a Bayesian posterior distribution if we are only allowed to touch a small mini-batch of data-items for every sample we generate?. An algorithm based on the Langevin equation with stochastic gradients (SGLD) was previously proposed to solv…
ReLU networks trained with MILPs match deep learning accuracy.
problem Training deep neural networks efficiently.
method Iterative training with Mixed Integer Linear Programs (MILPs).
result ReLU networks can be trained with MILPs achieving similar accuracy to deep learning methods.
This paper surveys distributed training techniques for deep learning models.
problem Substantial compute needed for training deep learning models.
method Exploration of various algorithms and techniques for distributed training.
result Recent advancements have reduced training time from weeks to minutes.
The paper provides concentration inequalities for Markov chain variance estimators.
problem Estimating the variance of Markov chains with concentration properties.
method Martingale decomposition method for uniformly geometrically ergodic Markov chains.
result Explicit control of the p-th moment of the OBM estimator difference and dependence on p and mixing time.
Bayesian optimization speeds up bioprocess development across scales.
problem Costly and complex bioprocess development across scales and biocatalyst selection.
method Multi-fidelity batch Bayesian optimization framework integrating Gaussian Processes and mixed-variable optimization.
result Reduction in experimental costs and increased yield in bioprocess optimization.
New method predicts spatio-temporal data with short and long-range dependence.
problem Uncertainty in predicting the distribution of mixed moving average fields.
method Theory-guided machine learning approach using generalized Bayesian algorithm.
result Fixed-time and any-time PAC Bayesian bounds for ensemble forecasts.
A robust approach compensates for small-data tasks in mixed linear regression.
problem Learning from small batches of data in tasks with many similar but insufficiently labeled examples.
method Spectral approach combining outlier-robust PCA and sum-of-squares algorithms.
result The approach achieves a graceful statistical trade-off, allowing smaller tasks than previously required.
This paper resolves the Langevin Algorithm's mixing time for log-concave distributions.
problem Resolving the mixing time of the Langevin Algorithm for log-concave sampling.
method Introducing Privacy Amplification by Iteration to analyze Rényi divergence and Optimal Transport smoothing.
result Optimal mixing bounds for the Langevin Algorithm in log-concave sampling settings.
Prediction-time batch normalization improves model robustness under covariate shift.
problem Dealing with covariate shift in deep learning models.
method Prediction-time batch normalization, a simple but effective method.
result Significantly improves model accuracy and calibration under covariate shift.
This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…
Self-augmentation improves deep networks for few-shot learning with minimal training data.
problem Improving deep networks' generalization to unseen classes with limited training examples.
method Self-augmentation using self-mix and self-distillation techniques, combined with regional dropout and local representation learning.
result The method outperforms state-of-the-art few-shot learning methods on prevalent benchmarks.
New technique reduces memory usage and boosts neural network training speed.
problem Training large neural networks requires significant memory and computational resources.
method L2L (layer-to-layer) execution technique with eager param-server (EPS) and micro-batching.
result 45% reduction in memory usage and 40% increase in throughput for BERT-Large.
Develops a contraction framework for MCMC mixing rates.
problem Proving mixing-time bounds for MCMC algorithms.
method Global and local contraction coefficients under Eγ-divergence. result Explicit global contraction coefficients for Gaussian smoothing.
ARMED models improve deep learning interpretability and generalize better on clustered data.
problem Clustered data leads to spurious associations and poor model fitting.
method Adversarial regularization and mixed effects subnetworks.
result ARMED models outperform conventional methods in accuracy and generalization.
Unified approach for first-order methods with Markovian noise in stochastic optimization and variational inequalities.
problem Stochastic optimization problems with Markovian noise.
method Unified theoretical analysis of first-order gradient methods using randomized batching and multilevel Monte Carlo.
result Optimal (linear) dependence on the mixing time of the noise sequence, eliminating previous limiting assumptions.
In this paper, we propose a novel technique to implement stochastic gradient methods, which are beneficial for learning from large datasets, through accelerated stochastic dynamics. A stochastic gradient method is based on mini-batch learning for reducing the computational cost when the amount of data is large. The sto…
Proposes an efficient algorithm for mHealth that makes real-time physical activity suggestions.
problem Training complex models efficiently in real-time mHealth applications.
method Streamlined empirical Bayes procedure for fitting linear mixed effects models.
result Improves accuracy and speed over state-of-the-art approaches in mHealth applications.
New method learns DAG structure in clustered data, accounting for local variations.
problem Learning DAG structure in clustered data with varying effects.
method Extends mixed models to structure learning, using a differentiable graph coupling mechanism.
result Asymptotically recovers true structure, detecting dependencies missed by other methods.
We improve MoE models for classification with rigorous guarantees and practical methods.
problem Limited guarantees for stable maximum-likelihood training and model selection in softmax-gated MoE models.
method Derived a batch MM algorithm with closed-form updates, proved finite-sample rates, and developed a dendrogram selector.
result Achieved near-parametric optimal rates for parameter recovery and improved accuracy over baselines.
The existence of a phase transition with diverging susceptibility in batch Minority Games (MGs) is the mark of informationally efficient regimes and is linked to the specifics of the agents' learning rules. Here we study how the standard scenario is affected in a mixed population game in which agents with the `optimal'…
New analysis shows SGD with noise doesn't leak more privacy with more iterations.
problem Privacy loss in noisy SGD with more iterations.
method Privacy Amplification by Iteration and Sampled Gaussian Mechanism.
result Privacy loss remains constant after a burn-in period, not increasing with more iterations.
RMGD uses bandit theory to optimize mini-batch size for faster and better performance.
problem Determining the optimal mini-batch size for gradient descent is time-consuming.
method Resilient Mini-batch Gradient Descent (RMGD) using Multi-Armed Bandit.
result RMGD achieves better performance than grid search in less time.
New research shows many batch selection methods for training work just as well as full batch training.
problem Finding optimal batch selection methods for training.
method Analysis of mini-batch Gradient Descent (GD) and Stochastic GD (SGD) with various batch selection rules.
result All mini-batch schedules, including deterministic ones, generalize optimally for smooth Lipschitz-convex/nonconvex/strongly-convex loss functions.
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
problem Batch Normalization's dependency on mini-batch elements can degrade performance for small batches.
method Filter Response Normalization (FRN) operates independently on each activation channel of each batch element.
result FRN layer outperforms BN and other alternatives in various settings for all batch sizes.
We analyze how batch learning impacts bandit problems.
problem Impact of batch learning in stochastic bandits.
method Policy-agnostic regret analysis, upper and lower bounds demonstration.
result The impact of batch learning can be measured in terms of online behavior.
Batch augmentation improves deep learning training by reducing batch size requirements.
problem Training deep neural networks with large batches can lead to overfitting.
method Replicate samples within a batch with different data augmentations.
result Batch augmentation reduces the number of necessary SGD updates for achieving the same accuracy.
AdaBatch dynamically adjusts batch size during training for deep learning models.
problem Choosing optimal batch size for deep neural networks.
method Adaptive batch size adjustment during training.
result Adaptive batch sizes improve performance by up to 6.25x on 4 GPUs with minimal accuracy loss.
BatchGFN uses generative flow networks for efficient batch active learning.
problem Efficiently selecting informative batches for active learning.
method Generative flow networks to sample batches proportional to a batch reward.
result Constructs highly informative batches with a single forward pass per point.
Wider neural networks perform better with large batches.
problem Communication overheads in small-batch training.
method Theoretical analysis and experiments on neural networks.
result Wider networks are more suitable for large-batch training.
Adaptive batch sizes improve active learning efficiency and flexibility.
problem Fixed batch sizes in active learning are inefficient due to dynamic cost-speed trade-offs.
method Probabilistic Numerics framework that adaptively changes batch sizes based on integration error and precision objectives.
result Significant enhancement in learning efficiency and flexibility across various applications.
Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.
problem Batched best arm identification (BBAI) problem, aiming to minimize policy switches and resource usage.
method Proposed Opt-BBAI algorithm, achieving near-optimal sample and batch complexity in non-asymptotic settings.
result First algorithm to achieve near-optimal sample and batch complexity in non-asymptotic settings.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
A new method designs batches for Bayesian optimization more efficiently.
problem Efficiently designing batches for Bayesian optimization to reduce total time.
method Minimal Terminal Variance (MTV) acquisition function, optimizing I-optimality criterion.
result MTV designs batches more efficiently than other methods, as shown by numerical experiments.
BaSE policy optimizes multi-armed bandits with batched data.
problem Optimizing multi-armed bandits with batched data.
method BaSE (batched successive elimination) policy for batched multi-armed bandits.
result Achieves rate-optimal regrets with adaptive batch sizes.
Four improvements to Batch Normalization improve deep learning performance.
problem Improving Batch Normalization for better deep learning performance.
method Proposed improvements include reasoning about current examples, Ghost Batch Normalization, weight decay regularization, and a new normalization algorithm for small batch sizes.
result Performance gains across all batch sizes with no additional computation during training.
Riemannian stochastic gradient descent converges faster with increasing batch size.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Theoretical analysis and numerical investigation of increasing batch size effects.
result Riemannian stochastic gradient descent converges faster with increasing batch size.
Polynomial-time algorithm for list-decodable linear regression with batches.
problem Efficiently decoding linear regression with a fraction of adversarial data.
method Polynomial time algorithm using batches of i.i.d. samples.
result Returns a list of size O(1/α^2) with one item close to true parameter.
New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.
problem Understanding the stability and convergence of mini-batch SGD.
method Analyzing the mini-batch Hessian and its directional curvature.
result Mini-batch SGD operates in a different stability regime (Edge of Stochastic Stability) compared to full-batch GD.