New algorithm picks multiple best points at once for faster optimization.
problem Optimizing multiple points simultaneously in noisy environments.
method Parallel Knowledge Gradient method for batch Bayesian optimization.
result Significantly faster at finding global optima compared to previous methods.
We develop parallel predictive entropy search (PPES), a novel algorithm for Bayesian optimization of expensive black-box objective functions. At each iteration, PPES aims to select a batch of points which will maximize the information gain about the global maximizer of the objective. Well known strategies exist for sug…
Bayesian optimization improved for high-dimensional problems through latent structure learning and parallel batched evaluations.
problem Challenges in optimizing high-dimensional black-box functions.
method Assuming a latent additive structure, using Gibbs sampling for structure learning, and determinantal point processes for batched queries.
result The proposed method outperforms existing approaches in both synthetic and real-world functions.
ε-shotgun optimizes expensive function evaluations in parallel.
problem Optimizing expensive black-box functions efficiently.
method ε-greedy batch Bayesian optimisation.
result ε-shotgun outperforms state-of-the-art batch methods.
Improves BO efficiency by allowing asynchronous parallel computing.
problem Wasteful use of resources in batch Bayesian optimisation.
method Developed PLAyBOOK for asynchronous local penalisation.
result Asynchronous BO often outperforms synchronous BO.
A scalable portfolio approach speeds up Bayesian optimization for noisy functions.
problem Efficiently selecting multiple designs in parallel for noisy, expensive black-box optimization.
method A portfolio approach that balances exploration and exploitation, using a scalable allocation strategy.
result Significant speed improvements over existing methods, with similar or better performance.
Parallel Gaussian process surrogate for noisy likelihood evaluations in Bayesian inference.
problem Bayesian inference with limited noisy log-likelihood evaluations from complex models.
method Hierarchical Gaussian process surrogate model for log-likelihood, batch-sequential design strategies.
result Robust, highly parallelizable, and sample-efficient method.
New method speeds up Bayesian optimization for parallel evaluations.
problem Efficiently parallelizing Bayesian optimization algorithms.
method Closed-form formula for a generalized expected improvement criterion.
result Derivation of a closed-form gradient for efficient batch optimization.
In this paper, we consider the challenge of maximizing an unknown function f for which evaluations are noisy and are acquired with high cost. An iterative procedure uses the previous measures to actively select the next estimation of f which is predicted to be the most useful. We focus on the case where the function ca…
SOBER optimizes and quadrates efficiently in parallel for diverse tasks.
problem Scalability of batch Bayesian optimization and quadrature for expensive functions.
method Reformulates batch selection as a quadrature problem, balancing exploitation and exploration.
result SOBER outperforms 11 baselines on 12 tasks.
Parallelizes active learning for Bayesian inference using Nested Sampler.
problem Expensive likelihood evaluations in complex experiments.
method Uses Nested Sampler to generate nearly-optimal batches of candidates in parallel.
result Comparable accuracy to sequential conditioning with efficient parallelization.
Parallel BO method for multi-objective optimization with constraints.
problem Optimizing multiple objectives under constraints with expensive evaluations.
method PPESMOC, a batch method for simultaneous optimization of black-box functions.
result Empirical evidence shows PPESMOC is effective for multi-objective optimization with constraints.
Adaptive batch size schedules improve language model training efficiency and generalization.
problem Dilemma of choosing batch sizes in large-scale model training.
method General-purpose adaptive batch size schedules compatible with data and model parallelism.
result Adaptive batch size schedules outperform constant batch sizes and heuristic warmup schedules.
A new method for parallelizing neural network training on large computers.
problem Efficiently parallelizing deep neural networks training on large distributed-memory computers.
method Integrates model, batch, and domain parallelism using a matrix-based parallel algorithm.
result Lowest communication costs achieved with an integrated approach, not pure model or data parallelism.
New batch learning framework improves scalability and accuracy of personalized ranking.
problem Inaccurate rank estimation in large-scale personalized ranking algorithms.
method Uses batch-based rank estimators and smooth rank-sensitive loss functions.
result Consistent accuracy improvements and time efficiency advantages over state-of-the-art methods.
SWAP uses large mini-batches to train DNNs faster with good generalization.
problem Training deep neural networks with small mini-batches is time-consuming.
method SWAP computes an approximate solution with large mini-batches and refines it by averaging weights of multiple parallel models.
result SWAP trains models as well as small-batch training but in significantly less time.
Bayesian optimization method for permutations accelerates combinatorial search.
problem Optimizing expensive-to-evaluate objectives on permutation problems.
method LAW2ORDER, a batch Bayesian optimization method based on the acquisition weighted kernel.
result LAW2ORDER achieves sublinear batch cumulative regret, demonstrating accelerated search.
ATS improves batch Bayesian Optimization by sampling multiple acquisition functions.
problem Efficiently optimizing multiple hyperparameters in parallel.
method ATS: sampling multiple acquisition functions from a stochastic process.
result ATS outperforms classical parallel Thompson Sampling and other batch BO methods.
New algorithm speeds up Gaussian process optimization.
problem Scalability issues in Gaussian process optimization.
method BBKB (Batch Budgeted Kernel Bandits) algorithm that selects candidates in batches and adaptively delays updates.
result Provable near-linear time complexity and significant speedup over existing methods.
FPDeep accelerates CNN training on FPGA clusters with high parallelism and energy efficiency.
problem Scaling DNN training to large clusters with high utilization and balanced workload.
method Hybrid model and layer parallelism, fine-grained pipeline, balanced workload partitioning.
result FPDeep achieves high parallelism and utilization, reducing storage demand to on-chip memory.
This paper improves a multi-batch L-BFGS method for machine learning.
problem Improving parallelization and stability in batch methods for machine learning.
method A multi-batch L-BFGS method with stable quasi-Newton updating.
result Shows how to perform stable quasi-Newton updating in the multi-batch setting.
Can one parallelize complex exploration exploitation tradeoffs? As an example, consider the problem of optimal high-throughput experimental design, where we wish to sequentially design batches of experiments in order to simultaneously learn a surrogate function mapping stimulus to response and identify the maximum of t…
A new method for efficient optimization of expensive simulations on HPC.
problem Efficiently solving computationally expensive simulation-based optimization problems.
method Asynchronous parallel Bayesian optimization with budgeted computational resource.
result Improved efficiency and handling of constraints in optimization.
Cyclic Data Parallelism reduces memory usage and balances gradient communications.
problem Training large deep learning models requires efficient parallelism to scale.
method Cyclic Data Parallelism shifts micro-batches from simultaneous to sequential execution, balancing memory and gradient communications.
result Cyclic Data Parallelism reduces total memory usage and balances gradient communications.
WMRB improves ranking accuracy and efficiency in scalable batch training.
problem Improving ranking accuracy and efficiency in large-scale recommendation systems.
method WMRB uses a new rank estimator and an efficient batch training algorithm.
result WMRB consistently outperforms WARP and other baselines in three item recommendation tasks.
New method ensures consistent inference across different tensor parallel sizes for large language models.
problem Non-deterministic inference in large language models due to inconsistent reduction orders across GPUs.
method Tree-Based Invariant Kernels (TBIK) that align intra- and inter-GPU reduction orders through a unified hierarchical binary tree structure.
result Bit-wise identical results across different tensor parallel sizes for RL training.
New framework extends linear scaling for kernel machines on GPUs.
problem Limited parallelization of large batch training on GPUs.
method Developed analytical framework for classical kernel machines.
result Extended linear scaling for faster training on GPUs.
Mesh-TensorFlow enables efficient deep learning on large clusters.
problem Memory constraints and inefficiency in batch-splitting for large models.
method Introduces Mesh-TensorFlow for specifying general tensor computations across a multi-dimensional mesh of processors.
result Trains Transformer models with up to 5 billion parameters on TPU meshes of up to 512 cores.
AdaBatch dynamically adjusts batch size during training for deep learning models.
problem Choosing optimal batch size for deep neural networks.
method Adaptive batch size adjustment during training.
result Adaptive batch sizes improve performance by up to 6.25x on 4 GPUs with minimal accuracy loss.
Parallelized bandit algorithms speed up decision-making.
problem Sequential exploration limits decision-making speed.
method Developed parallel contextual bandit algorithms.
result Regret nearly identical to sequential algorithms.
Many machine learning algorithms minimize a regularized risk, and stochastic optimization is widely used for this task. When working with massive data, it is desirable to perform stochastic optimization in parallel. Unfortunately, many existing stochastic optimization algorithms cannot be parallelized efficiently. In t…
Batch Thompson Sampling reduces exploration-exploitation trade-off in online decision making.
problem Balancing exploration and exploitation in online decision making.
method Introducing a batch Thompson Sampling framework for stochastic multi-arm bandit and linear contextual bandit problems.
result Achieves asymptotic regret bound with O(logT) batch queries, significantly reducing interactions. Pipelined Backpropagation trains large models without batches efficiently.
problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.
Study shows no degradation in neural network performance with larger batch sizes.
problem Characterizing the effects of increasing batch size on neural network training time.
method Experimentally measured training time for various neural network models and datasets.
result No evidence of degradation in out-of-sample performance with larger batch sizes.
This work analyzes mini-batching and averaging in SGD for least squares regression.
problem Improving parallelization and reducing variance in SGD for least squares regression.
method Characterizes mini-batching and tail-averaging for least squares regression, providing non-asymptotic excess risk bounds.
result Proves near-linear parallelization speedups and minimax risk for SGD with nearly the same number of updates as batch gradient descent.
K-FAC doesn't improve large batch training efficiency.
problem Inefficiency of K-FAC in large batch size training.
method Empirical analysis of K-FAC and SGD on ResNet and AlexNet.
result K-FAC doesn't exhibit improved scalability to large batch sizes.
New framework PBBO optimizes latent functions with preferential feedback.
problem Optimizing latent functions with parallel preferential feedback.
method Gaussian process model with a likelihood for parallel comparisons.
result Framework PBBO generalizes and expands Bayesian optimization.
Dynamic batching enables efficient training of graph neural networks.
problem Efficient training and inference of graph neural networks with dynamic computation graphs.
method Dynamic batching technique and high-level library for compositional blocks.
result Concise and batch-wise parallel implementations of dynamic graph models.
Increase batch size instead of learning rate for faster training.
problem Training time and parallelism improvements.
method Increase batch size during training.
result Equivalent test accuracies with fewer updates and shorter training times.
We determine the critical batch size for large language models and find it scales with data size, not model size.
problem Determining the optimal batch size for large-scale model training.
method We propose a measure of critical batch size, pre-trained models, and systematic hyper-parameter sweeps.
result The critical batch size scales primarily with data size, not model size.
A new parallel algorithm speeds up Hawkes process estimation.
problem Slow maximum likelihood estimation for Hawkes processes.
method Parallel prefix scan for sparse transition matrices.
result Massive speedup with O(N/P) complexity. The popularity of Bayesian optimization methods for efficient exploration of parameter spaces has lead to a series of papers applying Gaussian processes as surrogates in the optimization of functions. However, most proposed approaches only allow the exploration of the parameter space to occur sequentially. Often, it is…
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
This work deals with parallel optimization of expensive objective functions which are modeled as sample realizations of Gaussian processes. The study is formalized as a Bayesian optimization problem, or continuous multi-armed bandit problem, where a batch of q > 0 arms is pulled in parallel at each iteration. Several a…
Study the effects of data parallelism and sparsity on neural network training.
problem Understanding the effects of data parallelism and sparsity on neural network training.
method Conducted extensive experiments and developed a theoretical analysis.
result Found a general scaling trend between batch size and number of training steps to convergence for the effect of data parallelism, and difficulty of training under sparsity.
DBS dynamically adjusts batch sizes to improve cluster utilization.
problem Inefficient use of high-performance workers in synchronous distributed training.
method Dynamic Batch Size (DBS) strategy that adjusts batch size and dataset partition based on worker performance.
result Significantly improves cluster utilization and reduces training time.
AdaBatch speeds up SGD for sparse problems with minimal code changes.
problem Efficiently aggregating gradients in stochastic optimization for sparse problems.
method AdaBatch: a new aggregation operator for mini-batches.
result AdaBatch achieves similar sample efficiency to standard SGD but with larger batch sizes.
Efficiently optimize GPs by reusing candidate solutions multiple times.
problem High computational cost of Gaussian process optimization due to unique historical points.
method Sticking to a candidate solution for multiple evaluation steps and limiting switches.
result Improved efficiency and practicality of Gaussian process optimization algorithms.