BOE reformulates BO as a classifier for scalable batch optimisation.
problem Scalable batch optimisation of expensive functions.
method Reformulates BO as density-ratio estimation, removing need for explicit function prior.
result Theoretical guarantees and improved uncertainty estimates for batch optimisation.
SOBER optimizes and quadrates efficiently in parallel for diverse tasks.
problem Scalability of batch Bayesian optimization and quadrature for expensive functions.
method Reformulates batch selection as a quadrature problem, balancing exploitation and exploration.
result SOBER outperforms 11 baselines on 12 tasks.
New method optimizes multiple points in Bayesian optimization efficiently.
problem Optimizing multiple points in expensive black-box functions.
method Reformulated BO as probability measure optimization, using convex gradient flows.
result Demonstrated effectiveness on various benchmarks compared to state-of-the-art methods.
Batch Bayesian optimisation (BO) has been successfully applied to hyperparameter tuning using parallel computing, but it is wasteful of resources: workers that complete jobs ahead of others are left idle. We address this problem by developing an approach, Penalising Locally for Asynchronous Bayesian Optimisation on k…
GIBBON unifies Bayesian optimization for various problem types.
problem Bayesian optimization across noisy, multi-fidelity, and batch optimisations.
method Novel approximation of information gain for a single framework.
result Significantly lower computational overhead compared to existing approaches.
Efficient optimisation of black-box problems that comprise both continuous and categorical inputs is important, yet poses significant challenges. We propose a new approach, Continuous and Categorical Bayesian Optimisation (CoCaBO), which combines the strengths of multi-armed bandits and Bayesian optimisation to select …
This paper tackles batch Bayesian optimal experimental design by using Wasserstein gradient flows.
problem The challenge of optimising high-dimensional, strongly non-convex expected information gain in batch settings.
method Probabilistic lifting to the space of probability measures, entropic regularisation, Wasserstein gradient flow, and particle-based algorithms.
result The proposed approach can be used directly as a randomised batch-design policy or as a computational relaxation.
Bayesian optimisation is a popular, surrogate model-based approach for optimising expensive black-box functions. Given a surrogate model, the next location to expensively evaluate is chosen via maximisation of a cheap-to-query acquisition function. We present an ε-greedy procedure for Bayesian optimisation in batch s…
SOBER framework optimizes Bayesian optimization tasks efficiently.
problem Challenges in parallel Bayesian optimization.
method Probabilistic Lifting with Kernel Quadrature.
result Versatile and flexible batch Bayesian optimization.
New method reduces variance in complex probabilistic model optimization.
problem High variance in stochastic optimisation of complex models.
method Use recognition network to approximate optimal control variate for each mini-batch.
result Sub-optimal variance reduction is improved with new approach.
Bayesian optimisation (BO) is widely used to optimise stochastic black box functions. While most BO approaches focus on optimising conditional expectations, many applications require risk-averse strategies and alternative criteria accounting for the distribution tails need to be considered. In this paper, we propose ne…
Extends hyperparameter transfer across model sizes and modules, improving training speed.
problem Training stability and performance of large-scale models with optimal hyperparameters.
method Complete(d) Parameterisation, per-module hyperparameter optimisation and transfer. result Hyperparameter transfer holds even in the per-module hyperparameter regime, improving training speed.
AEGiS optimizes expensive function evaluations asynchronously.
problem Optimizing expensive black-box functions efficiently.
method Asynchronous ε-Greedy Bayesian Optimisation combining greedy search and Thompson sampling.
result AEGiS outperforms existing asynchronous BO methods.
Integration over non-negative integrands is a central problem in machine learning (e.g. for model averaging, (hyper-)parameter marginalisation, and computing posterior predictive distributions). Bayesian Quadrature is a probabilistic numerical integration technique that performs promisingly when compared to traditional…
Deep Neural Network (DNN) acoustic models often use discriminative sequence training that optimises an objective function that better approximates the word error rate (WER) than frame-based training. Sequence training is normally implemented using Stochastic Gradient Descent (SGD) or Hessian Free (HF) training. This pa…
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
MTFL improves UA and speeds convergence in personalised DNNs on edge devices.
problem Non-IID user data harms FL convergence and global UA is not always the goal.
method Introduces non-federated BN layers into federated DNNs for personalised training.
result MTFL reduces UA rounds by up to 5x and convergence time by up to 3x.
HydaLearn dynamically adjusts task weights for better MTL performance.
problem Constant loss weights in MTL lead to poor results due to drifting relevance and varying mini-batch composition.
method HydaLearn uses mini-batch gradients to dynamically adjust task weights.
result HydaLearn improves performance on synthetic and real-world data.
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch n…
New algorithms minimize MMD to approximate probability measures efficiently.
problem Approximating probability measures by representative point sets.
method Sequential greedy minimization of maximum mean discrepancy (MMD) over candidate sets, with mini-batch variants.
result Consistency of proposed algorithms and mini-batch variants established.
New minibatching strategy reduces stochastic gradient bias in optimisation.
problem Reducing bias in stochastic gradient descent.
method Symmetric Minibatching Strategy combined with momentum.
result Reduced stochastic gradient bias from O(h2) to O(h4). ARC algorithm optimizes dynamic pricing with correlated observations.
problem Optimizing dynamic pricing with correlated and generally distributed observations.
method Extends ARC algorithm to batched bandits with generalised linear model.
result ARC algorithm outperforms alternative approaches in dynamic pricing.
Novel approach finds implicit regularisation in two-player games using BEA.
problem Understanding implicit regularisation in two-player games.
method Using backward error analysis to construct continuous-time flows with gradient-eligible vector fields.
result Identifies new implicit regularisation effects in two-player games.
New insights into learning rates and batch sizes for neural networks using random matrix theory.
problem Understanding how batch size affects learning rates in neural networks.
method Random matrix theory applied to spiked, field-dependent random matrices.
result Analytical expressions for maximal learning rates as a function of batch size.
We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly …
Warm starts improve Gaussian process regression by up to 16x.
problem Optimizing hyperparameters for Gaussian processes.
method Iterative Gaussian processes with warm start optimization.
result Warm starts achieve the same results as conventional methods but significantly speed up computations.
The Extreme Deconvolution method fits a probability density to a dataset where each observation has Gaussian noise added with a known sample-specific covariance, originally intended for use with astronomical datasets. The existing fitting method is batch EM, which would not normally be applied to large datasets such as…
Revisits online Laplace methods for neural networks, showing they are sound under certain conditions.
problem Online Laplace methods violate the Laplace approximation's critical assumption.
method Re-derives online Laplace methods, showing they target a variational bound on a mode-corrected variant of the Laplace evidence.
result Online Laplace and its mode-corrected counterpart share stationary points that satisfy the Laplace method's assumption.
Online learning rbfnet improves multi-horizon returns forecasts for financial time series.
problem Nonstationarity and concept drift in financial time series.
method Combines feature representation transfer with sequential optimisation.
result Online learning rbfnet outperforms random-walk and batch learners.
The computational efficiency of approximate Bayesian computation (ABC) has been improved by using surrogate models such as Gaussian processes (GP). In one such promising framework the discrepancy between the simulated and observed data is modelled with a GP which is further used to form a model-based estimator for the …
This paper explores optimising acquisition functions in Bayesian optimisation.
problem Optimising acquisition functions in Bayesian optimisation is challenging due to their non-convex nature.
method The authors derive compositional forms for acquisition functions and use them to recast maximisation as a compositional optimisation problem.
result The compositional approach to maximising acquisition functions shows empirical advantages across various tasks.
New algorithm optimizes auction prices in real-time.
problem Maximizing revenue in online auctions with high frequency data.
method First real-time algorithm for online learning of monopoly prices.
result Achieves constant time and memory complexity for updates.
Bayesian Predictive Coding improves deep learning uncertainty quantification.
problem Limitations of maximum a posteriori and maximum likelihood estimates in predictive coding.
method Developed Bayesian Predictive Coding (BPC) that estimates a posterior distribution over network parameters.
result BPC offers comparable uncertainty quantification to existing methods in Bayesian deep learning and improves convergence properties.
New framework converts multi-objective to single-objective optimisation.
problem Solving multi-objective optimisation problems.
method Formalises scalarisation into mathematical framework, uses R2 utilities.
result R2 utilities are monotone and submodular, optimised by greedy algorithms.
Unified Bayesian Optimisation for mixed variables improves performance.
problem Efficient optimisation of problems with both categorical and continuous variables.
method Derive value proposals from the Expected Improvement criterion to optimise both categorical and continuous variables under a single acquisition metric.
result Unified approach significantly outperforms existing methods across mixed-variable tasks.
Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima. In this work we develop a similar correspondence for minibatch natural gradient descent (NGD). We prove that for sufficiently small learning rates, if the model predictions on the trai…
A new method learns priors for Bayesian optimisation to improve performance.
problem Bayesian optimisation tasks often assume strong similarity, which is violated in many cases.
method Replace strong similarity assumption with shape similarity, learn priors for hyperparameters.
result PLeBO and prior transfer find good inputs in fewer evaluations.
Study improves Bayesian optimisation with ensemble transfer learning.
problem Improving sample efficiency in Bayesian optimisation of expensive functions.
method Empirical analysis of ensemble-based transfer learning methods and pipeline components.
result Two components (warm start initialisation and positive weight constraint) improve transfer learning Bayesian optimisation performance.
AdamZ optimiser improves neural network training efficiency.
problem Challenges in optimisation like overshooting and stagnation.
method Dynamic learning rate adjustment based on overshoot and stagnation factors.
result Consistently minimises loss function, improving model performance.
Wind farm layout optimisation tackles space constraints with Bayesian multi-objective approach.
problem Optimizing wind farm layout due to limited space and conflicting objectives.
method Set-based multi-objective Bayesian optimisation using Gaussian process.
result Demonstrates potential of set-based Bayesian multi-objective optimisation for wind farm layout.
Book covers tools for zeroth-order convex optimisation.
problem Zeroth-order convex optimisation.
method Cutting plane methods, interior point methods, continuous exponential weights, gradient descent, online Newton step.
result Improved existing bounds and algorithms.
Bayesian optimisation tackles high-dimensional categorical and mixed search spaces.
problem Bayesian optimisation on high-dimensional categorical and mixed search spaces is challenging.
method Combining local optimisation with a tailored kernel design.
result Empirically outperforms current baselines in performance and computational costs.
O3 optimizes samples from generative models without extra training.
problem Optimizing specific criteria within rich data distributions.
method Surrogate latent spaces for efficient black-box optimisation.
result Surrogate-space optimisation finds higher-scoring samples.
A new optimisation method efficiently scales Hessian-vector products for neural networks.
problem Challenges in applying second-order quasi-Newton methods due to large Hessian and non-convexity.
method Proposes an optimisation algorithm that asymptotically uses the exact inverse Hessian with modified eigenvalues.
result Demonstrates scalability and comparable performance to other optimisation methods in neural networks.
Bayesian optimisation framework for graph functions.
problem Optimizing functions on graph structures efficiently.
method Learning suitable kernels and local modelling.
result Demonstrated effectiveness on synthetic and real-world graphs.
In this paper, we discuss software design issues related to the development of parallel computational intelligence algorithms on multi-core CPUs, using the new Java 8 functional programming features. In particular, we focus on probabilistic graphical models (PGMs) and present the parallelisation of a collection of algo…
Bayesian optimisation tackles expensive black-box functions with constraints.
problem Optimizing constrained black-box functions in machine learning and simulation.
method Proposes a new Knowledge Gradient acquisition function for constrained Bayesian optimisation.
result Demonstrates superior performance over four state-of-the-art constrained Bayesian optimisation algorithms.
MTL2L learns to adapt optimisation rules for unseen data.
problem Learners need to adapt to unseen data domains.
method Introduces MTL2L, a context-aware neural optimiser.
result MTL2L can adapt optimisation rules for unseen data.