A new batch construction method for RNNs outperforms existing approaches in MXNet.
problem Improving the efficiency and performance of recurrent neural networks in MXNet.
method Proposes an alternately sorted batch construction strategy for RNNs.
result Alternately sorted batches outperform bucketing and other methods in training time and recognition performance.
Paper develops a method to construct confidence regions for model parameters using batch means method.
problem Constructing confidence regions for model parameters in stochastic gradient descent.
method Batch means method to cancel out covariance matrix, using Polyak-Ruppert averaging.
result Established process-level functional central limit theorem for stochastic gradient descent estimators.
BatchGFN uses generative flow networks for efficient batch active learning.
problem Efficiently selecting informative batches for active learning.
method Generative flow networks to sample batches proportional to a batch reward.
result Constructs highly informative batches with a single forward pass per point.
Adaptive batch sizes improve active learning efficiency and flexibility.
problem Fixed batch sizes in active learning are inefficient due to dynamic cost-speed trade-offs.
method Probabilistic Numerics framework that adaptively changes batch sizes based on integration error and precision objectives.
result Significant enhancement in learning efficiency and flexibility across various applications.
Matching pursuit (MP) methods are a promising class of feature construction algorithms for value function approximation. Yet existing MP methods require creating a pool of potential features, mandating expert knowledge or enumeration of a large feature pool, both of which hinder scalability. This paper introduces batch…
A new method designs batches for Bayesian optimization more efficiently.
problem Efficiently designing batches for Bayesian optimization to reduce total time.
method Minimal Terminal Variance (MTV) acquisition function, optimizing I-optimality criterion.
result MTV designs batches more efficiently than other methods, as shown by numerical experiments.
A new method for graph neural networks speeds up inference and training.
problem Challenges in constructing mini-batches for large graphs in graph neural networks.
method Theoretical model of batch construction via maximizing influence score of nodes on outputs.
result Accelerates inference by up to 130x compared to previous methods.
New algorithm picks multiple best points at once for faster optimization.
problem Optimizing multiple points simultaneously in noisy environments.
method Parallel Knowledge Gradient method for batch Bayesian optimization.
result Significantly faster at finding global optima compared to previous methods.
Proposes qPO, a new acquisition strategy for batched Bayesian optimization that maximizes the probability of including the optimum.
problem Efficiently identifying top-performing compounds from a large chemical library.
method qPO (multipoint Probability of Optimality) acquisition strategy that maximizes the probability of including the true optimum.
result Empirical evidence shows that qPO is competitive with and complements other state-of-the-art methods in batched Bayesian optimization.
New method optimizes multiple points in Bayesian optimization efficiently.
problem Optimizing multiple points in expensive black-box functions.
method Reformulated BO as probability measure optimization, using convex gradient flows.
result Demonstrated effectiveness on various benchmarks compared to state-of-the-art methods.
New method for constructing confidence intervals for time series data.
problem Constructing confidence intervals for statistical functionals from time series data.
method Proposes a general purpose confidence interval procedure based on overlapping batches of time series data.
result Large overlapping batches yield confidence intervals of higher quality than generic methods.
A new method for batch prediction sets in classification problems.
problem Constructing reliable prediction sets for multiple unlabeled examples.
method Proposes a uniformly more powerful approach to batch prediction sets using specific combinations of conformal p-values.
result The proposed method provides narrower prediction sets compared to the Bonferroni correction.
A new MH-MCMC method using mini-batches and stochastic gradient for scalable inference.
problem Computational inefficiency of traditional MCMC algorithms for large datasets.
method Mini-batch MH-MCMC with reversible stochastic gradient proposal.
result The method provides approximate tempered stationary distribution and reasonable acceptance probabilities.
Recency Bias selects recent uncertain samples for faster, more accurate deep learning.
problem Improving the accuracy of deep neural networks through better mini-batch selection.
method Uses historical label predictions to evaluate predictive uncertainty and selects samples proportionally.
result Reduces test error by up to 20.97% compared to existing methods in the same training time.
Bayesian batch active learning approximates model parameters efficiently.
problem High label acquisition cost for large-scale supervised models.
method Sparse subset approximation using Bayesian active learning.
result Efficient active learning at scale with diverse batches.
Proposes a new batch selection method for multi-label classification.
problem Improving the accuracy of deep neural networks in multi-label classification tasks.
method Adapts uncertainty measures to multi-label data, considering label correlations and dynamic uncertainty.
result Improves performance and accelerates convergence of multi-label deep learning models.
Breaks the hardness conjecture for batch RL with a novel tournament-based approach.
problem Sample-efficient reinforcement learning from exploratory data.
method BVFT algorithm using pairwise comparison and state-action partition.
result Solves the learning problem in a setting previously thought impossible.
Unified framework for high-dimensional online learning with non-divergent error bounds and adaptive gains.
problem Divergence of error bounds in high-dimensional online learning as data batches increase.
method Asynchronous decomposition framework with summary statistics and dynamic regularization.
result Non-divergent error bounds and adaptive gains in sparse online optimization.
Adam's bias shifts from full-batch to max-margin of different norms for separable data.
problem Understanding Adam's implicit bias in the incremental batch setting.
method Analyzing incremental Adam on linearly separable data, constructing datasets, and using a proxy algorithm.
result Incremental Adam can converge to different max-margin classifiers depending on the dataset and batching scheme.
Generative models enhance BO for large batch optimization.
problem Efficiently sampling solutions in high-dimensional, combinatorial design spaces.
method Train generative models to sample solutions proportional to expected utility.
result Generative models can approximate optimal target distributions under certain conditions.
New study analyzes security of neural network data reconstruction attacks.
problem Data reconstruction attacks pose a threat to private training data.
method Analyzes security boundary of data reconstruction attacks via neuron exclusivity state.
result Characterizes insecure/secure boundary of data reconstruction attacks.
Bayesian optimization improved for high-dimensional problems through latent structure learning and parallel batched evaluations.
problem Challenges in optimizing high-dimensional black-box functions.
method Assuming a latent additive structure, using Gibbs sampling for structure learning, and determinantal point processes for batched queries.
result The proposed method outperforms existing approaches in both synthetic and real-world functions.
Deep learning uses complex networks for high-dimensional data.
problem Computational inefficiency in training deep learning models.
method Use of hierarchical latent variables, efficient linear algebra, SGD optimization, and batch sampling.
result Efficient training and inference possible with optimized algorithms.
Study examines line search approximations for neural networks using MBSS.
problem Reducing computational cost in training large-scale neural networks.
method Empirical study of quadratic line search approximations for dynamic MBSS loss functions, enforcing different types of function and derivative information.
result Selectively enforcing information in approximations reduces the variance of predicted step sizes.
Paper addresses OPE for dependent bandit samples using MDS and batch updates.
problem Evaluating policies from non-i.i.d. historical data in contextual bandits.
method Constructs an MDS-based estimator for dependent samples, solves batch update and deficient support issues.
result Derives an asymptotically normal estimator for evaluation policy value.
New RL difficulty shown for discounted settings.
problem Difficulty in reinforcement learning with discounted rewards.
method Adapted Wang et al. (2020) construction to 2-state MDP.
result Learning impossible even with infinite data in discounted setting.
A new inference method using regression and batched discrepancies.
problem Simulating parameters from simulator outputs.
method Regression-based projection and batched discrepancy weighting.
result Method produces a self-normalized pseudo-posterior.
Improved deep probabilistic time series forecasting by learning error autocorrelation.
problem Simplification of time-independent error process and lack of serial correlation in existing models.
method Proposes a training method that incorporates error autocorrelation to enhance probabilistic forecasting accuracy.
result Improves predictive accuracy and uncertainty quantification across multiple datasets.
Ensembles of randomized decision trees, usually referred to as random forests, are widely used for classification and regression tasks in machine learning and statistics. Random forests achieve competitive predictive performance and are computationally efficient to train and test, making them excellent candidates for r…
BMBO-DARN optimizes expensive functions with varying fidelities.
problem Optimizing expensive, multi-fidelity functions efficiently.
method Batch Multi-fidelity Bayesian Optimization with Deep Auto-Regressive Networks.
result BMBO-DARN improves surrogate learning and optimization performance.
Framework aligns datasets using harmonic expansion of intrinsic geometry.
problem Combining datasets from different modalities or correcting batch effects.
method Alignment through harmonic expansion of diffusion coordinates.
result Unified diffusion geometry for fused or corrected data.
ADPSGD enables faster ASR training with larger batch sizes.
problem Efficient distributed deep learning for ASR with large batch sizes.
method Asynchronous Decentralized Parallel Stochastic Gradient Descent (ADPSGD) and Hierarchical-ADPSGD (H-ADPSGD).
result ADPSGD can converge with a 3X larger batch size than SSGD, enabling faster training.
This paper introduces sample-averaged Q-learning for better RL performance.
problem Improving reinforcement learning algorithms by managing uncertainty.
method Integrates statistical inference into Q-learning through sample averaging and functional central limit theorem.
result Establishes a unified theoretical foundation for sample-averaged Q-learning.
RMGD uses bandit theory to optimize mini-batch size for faster and better performance.
problem Determining the optimal mini-batch size for gradient descent is time-consuming.
method Resilient Mini-batch Gradient Descent (RMGD) using Multi-Armed Bandit.
result RMGD achieves better performance than grid search in less time.
New research shows many batch selection methods for training work just as well as full batch training.
problem Finding optimal batch selection methods for training.
method Analysis of mini-batch Gradient Descent (GD) and Stochastic GD (SGD) with various batch selection rules.
result All mini-batch schedules, including deterministic ones, generalize optimally for smooth Lipschitz-convex/nonconvex/strongly-convex loss functions.
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
problem Batch Normalization's dependency on mini-batch elements can degrade performance for small batches.
method Filter Response Normalization (FRN) operates independently on each activation channel of each batch element.
result FRN layer outperforms BN and other alternatives in various settings for all batch sizes.
We analyze how batch learning impacts bandit problems.
problem Impact of batch learning in stochastic bandits.
method Policy-agnostic regret analysis, upper and lower bounds demonstration.
result The impact of batch learning can be measured in terms of online behavior.
This work tackles sequential data learning challenges by improving neural network robustness to non-iid distribution shifts.
problem Sequential data learning challenges, particularly non-iid distribution shifts across batches.
method Cramér-Rao-based regularization using Fisher Information Matrix to adapt to sequential covariate shifts.
result Achieves 19% accuracy improvement over state-of-the-art methods.
Batch augmentation improves deep learning training by reducing batch size requirements.
problem Training deep neural networks with large batches can lead to overfitting.
method Replicate samples within a batch with different data augmentations.
result Batch augmentation reduces the number of necessary SGD updates for achieving the same accuracy.
AdaBatch dynamically adjusts batch size during training for deep learning models.
problem Choosing optimal batch size for deep neural networks.
method Adaptive batch size adjustment during training.
result Adaptive batch sizes improve performance by up to 6.25x on 4 GPUs with minimal accuracy loss.
A new perfectly truthful calibration measure improves prediction reliability.
problem Improving the reliability of predictions by ensuring they are conditionally unbiased.
method Designing a simple, perfectly truthful calibration measure called ATB.
result ATB is the first perfectly truthful calibration measure in the batch setting.
Wider neural networks perform better with large batches.
problem Communication overheads in small-batch training.
method Theoretical analysis and experiments on neural networks.
result Wider networks are more suitable for large-batch training.
Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.
problem Batched best arm identification (BBAI) problem, aiming to minimize policy switches and resource usage.
method Proposed Opt-BBAI algorithm, achieving near-optimal sample and batch complexity in non-asymptotic settings.
result First algorithm to achieve near-optimal sample and batch complexity in non-asymptotic settings.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
Introduces a differentiable approximation to the zero-one loss.
problem Incompatibility of zero-one loss with gradient-based optimization.
method Smooth projection onto hypersimplex through constrained optimization.
result Achieves significant improvements in generalization under large-batch training.
BaSE policy optimizes multi-armed bandits with batched data.
problem Optimizing multi-armed bandits with batched data.
method BaSE (batched successive elimination) policy for batched multi-armed bandits.
result Achieves rate-optimal regrets with adaptive batch sizes.
New method reduces parameter overhead for Bayesian neural networks.
problem High parameter overhead and difficulty of implementation in variational Bayesian neural networks.
method Constructs a general variational family for ensemble-based Bayesian neural networks that works well with batch normalization layers.
result Improves predictive accuracy and achieves almost perfect calibration on a ResNet-18 trained with ImageNet.
A fast kernel method for efficient batch and online learning.
problem Efficient kernel methods for both batch and online learning.
method Random feature approximation using Mondrian trees.
result Fast kernel-width selection and efficient feature reuse.