Improves meta-learning efficiency with mixed-mode differentiation.
problem Efficiently calculating complex derivatives in meta-learning.
method Mixed-Flow Meta-Gradients (MixFlow-MG) for scalable differentiation.
result Significant memory and time improvements in meta-learning tasks.
Gradient learning optimises MCMC proposal distributions.
problem Intractable targets in MCMC sampling.
method Gradient-based optimisation of proposal distributions using a maximum entropy regularised objective function.
result Our method can outperform traditional MCMC algorithms, including Hamiltonian Monte Carlo.
Despite recent innovations in network architectures and loss functions, training RNNs to learn long-term dependencies remains difficult due to challenges with gradient-based optimisation methods. Inspired by the success of Deep Neuroevolution in reinforcement learning (Such et al. 2017), we explore the use of gradient-…
It seems to be a pearl of conventional wisdom that parameter learning in deep sum-product networks is surprisingly fast compared to shallow mixture models. This paper examines the effects of overparameterization in sum-product networks on the speed of parameter optimisation. Using theoretical analysis and empirical exp…
Unified framework for lifted training and inversion of neural networks.
problem Challenges in gradient-based training of deep neural networks.
method Unified framework encapsulating various lifted training strategies.
result Unified framework improves training landscape and stability.
This paper considers extractive summarisation in a comparative setting: given two or more document groups (e.g., separated by publication time), the goal is to select a small number of documents that are representative of each group, and also maximally distinguishable from other groups. We formulate a set of new object…
New methods speed up fitting for large datasets with noisy observations.
problem Fitting large datasets with Gaussian noise and known covariance.
method Two minibatch variants of extreme deconvolution, online EM algorithm, and gradient-based optimisation.
result Methods can scale to larger models and fit larger datasets faster.
One-pass optimisation for high-dimensional hyperparameters.
problem Efficient optimisation of hyperparameters in machine learning models.
method Approximate hypergradient-based optimisation for any continuous hyperparameter, requiring only one training episode.
result Competitive performance on various datasets without hyperparameter restarts.
LOGAN optimizes GAN training by improving adversarial dynamics.
problem Training GANs is challenging due to delicate adversarial dynamics and potential divergence.
method Integrates natural gradient-based latent optimisation into CS-GAN.
result Significant improvement in GAN training performance, achieving state-of-the-art results.
Optimising discrete data for a desired characteristic using gradient-based methods involves projecting the data into a continuous latent space and carrying out optimisation in this space. Carrying out global optimisation is difficult as optimisers are likely to follow gradients into regions of the latent space that the…
Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods …
Bayesian optimisation generates saliency maps for black-box models.
problem Generating saliency maps for models without access to parameters.
method Bayesian optimisation sampling method to find global salient regions.
result Approach outperforms grid-based methods and performs similarly to gradient-based methods.
MPLP learns neural network weights by treating operations as message-passing agents.
problem Training neural networks using gradient-based methods.
method MPLP abstracts neural network operations as message-passing agents, updating internal states and passing messages.
result MPLP outperforms traditional gradient-based methods on simple feed-forward neural networks.
Investigates gradient descent dynamics and introduces new regularisation methods.
problem Understanding and mitigating gradient descent instabilities and interactions with smoothness regularisation.
method Derives continuous-time flows to account for discretisation drift, constructs learning rate schedules and regularisers.
result New regularisation methods improve performance in reinforcement learning.
Adaptive model learns from time series data with changing distributions.
problem Predicting time series data under distribution shift.
method Formulates distribution shift as weighted empirical risk minimization. Uses a gradient-based learning method for a forgetting mechanism.
result Proposes an efficient method for adaptive time series prediction.
In many contexts the modal properties of a structure change, either due to the impact of a changing environment, fatigue, or due to the presence of structural damage. For example during flight, an aircraft's modal properties are known to change with both altitude and velocity. It is thus important to quantify these cha…
PDNAS optimizes GNN architectures for diverse datasets.
problem Inadequate adaptability and combinatorial search space in GNNs.
method Dual architecture search (micro- and macro-architectures) with gradient-based optimization.
result PDNAS finds deeper GNNs with better performance on diverse datasets.
Novel approach finds implicit regularisation in two-player games using BEA.
problem Understanding implicit regularisation in two-player games.
method Using backward error analysis to construct continuous-time flows with gradient-eligible vector fields.
result Identifies new implicit regularisation effects in two-player games.
During recent years there has been an increased interest in stochastic adaptations of limited memory quasi-Newton methods, which compared to pure gradient-based routines can improve the convergence by incorporating second order information. In this work we propose a direct least-squares approach conceptually similar to…
Implicit models, which allow for the generation of samples but not for point-wise evaluation of probabilities, are omnipresent in real-world problems tackled by machine learning and a hot topic of current research. Some examples include data simulators that are widely used in engineering and scientific research, genera…
Optimizes Euclidean functions on Riemannian manifolds with warped metrics.
problem Optimizing functions in high-dimensional Euclidean spaces.
method Riemannian geometry, warped metric, geodesic curves, Taylor approximations, retraction maps.
result Efficient optimization of functions using third-order approximations of geodesics.
Develops a kernel-based framework for dynamic trading strategies.
problem Optimizing portfolios with temporal dependencies in asset dynamics.
method Parameterizes trading strategies as functions in RKHS, enabling flexible, non-Markovian approaches.
result Significantly outperforms classical Markovian methods in synthetic and market-data examples.
The goal of reinforcement learning algorithms is to estimate and/or optimise the value function. However, unlike supervised learning, no teacher or oracle is available to provide the true value function. Instead, the majority of reinforcement learning algorithms estimate and/or optimise a proxy for the value function. …
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives. We derive an objective that, under automatic differentiation, produces low-variance unbiased estimators of derivatives at any order. Our objective is compatible with arbit…
AdamQLR optimizes Adam with K-FAC heuristics, achieving comparable performance to tuned benchmarks.
problem Improving the performance of Adam optimizers with stabilizing heuristics.
method Combining Adam's update directions with K-FAC's heuristics (damping and learning rate selection).
result Untuned AdamQLR can achieve comparable performance to tuned benchmarks.
Wide neural networks with asymmetrical node scaling converge globally and learn features.
problem Global convergence and feature learning in over-parameterised shallow networks.
method Gradient-based optimisation of wide, shallow neural networks with asymmetrical node scaling.
result Gradient flow and gradient descent converge to a global minimum and learn features, unlike in the NTK parameterisation.
Improved SSD for faster and more accurate goodness-of-fit tests and model learning.
problem Optimal slicing directions for SSD are computationally expensive and sub-optimal.
method Relaxed optimal slicing requirement, active sub-space construction, spectral decomposition.
result 14-80x speed-up in goodness-of-fit tests compared to gradient-based alternatives.
This paper uses Nested Sampling to improve Gaussian Process uncertainty quantification.
problem Underestimating predictive uncertainty and overfitting in Gaussian Process models.
method Marginalises hyperparameters using Nested Sampling for spectral mixture kernels.
result Improves predictive performance and uncertainty quantification across various data sets.
This paper explores optimising acquisition functions in Bayesian optimisation.
problem Optimising acquisition functions in Bayesian optimisation is challenging due to their non-convex nature.
method The authors derive compositional forms for acquisition functions and use them to recast maximisation as a compositional optimisation problem.
result The compositional approach to maximising acquisition functions shows empirical advantages across various tasks.
A new estimator improves training of probabilistic models with latent Gaussian variables.
problem Improving gradient estimation for models with latent Gaussian variables.
method Rao-Blackwellised Reparameterisation Gradients (R2-G2)
result R2-G2 consistently yields better performance in models with multiple applications of the reparameterisation trick.
New framework converts multi-objective to single-objective optimisation.
problem Solving multi-objective optimisation problems.
method Formalises scalarisation into mathematical framework, uses R2 utilities.
result R2 utilities are monotone and submodular, optimised by greedy algorithms.
Unified Bayesian Optimisation for mixed variables improves performance.
problem Efficient optimisation of problems with both categorical and continuous variables.
method Derive value proposals from the Expected Improvement criterion to optimise both categorical and continuous variables under a single acquisition metric.
result Unified approach significantly outperforms existing methods across mixed-variable tasks.
We solve 6-DoF localisation and 3D reconstruction using deep state-space models.
problem 6-DoF localisation and dense 3D reconstruction in spatial environments.
method Approximate Bayesian inference in a deep state-space model combining learning and domain knowledge.
result Near state-of-the-art performance on UAV flight data.
A new method learns priors for Bayesian optimisation to improve performance.
problem Bayesian optimisation tasks often assume strong similarity, which is violated in many cases.
method Replace strong similarity assumption with shape similarity, learn priors for hyperparameters.
result PLeBO and prior transfer find good inputs in fewer evaluations.
Gaussian processes (GPs) are Bayesian nonparametric generative models that provide interpretability of hyperparameters, admit closed-form expressions for training and inference, and are able to accurately represent uncertainty. To model general non-Gaussian data with complex correlation structure, GPs can be paired wit…
Enhanced feature learning using neural networks and kernel methods with improved robustness.
problem Improving feature learning and function estimation in supervised learning.
method Regularised empirical risk minimisation with a new kernel approach.
result The proposed method, BKerNN, converges to the minimal risk with explicit high-probability rates.
We present a probabilistic framework for both (i) determining the initial settings of kernel adaptive filters (KAFs) and (ii) constructing fully-adaptive KAFs whereby in addition to weights and dictionaries, kernel parameters are learnt sequentially. This is achieved by formulating the estimator as a probabilistic mode…
Study improves Bayesian optimisation with ensemble transfer learning.
problem Improving sample efficiency in Bayesian optimisation of expensive functions.
method Empirical analysis of ensemble-based transfer learning methods and pipeline components.
result Two components (warm start initialisation and positive weight constraint) improve transfer learning Bayesian optimisation performance.
AdamZ optimiser improves neural network training efficiency.
problem Challenges in optimisation like overshooting and stagnation.
method Dynamic learning rate adjustment based on overshoot and stagnation factors.
result Consistently minimises loss function, improving model performance.
Wind farm layout optimisation tackles space constraints with Bayesian multi-objective approach.
problem Optimizing wind farm layout due to limited space and conflicting objectives.
method Set-based multi-objective Bayesian optimisation using Gaussian process.
result Demonstrates potential of set-based Bayesian multi-objective optimisation for wind farm layout.
Book covers tools for zeroth-order convex optimisation.
problem Zeroth-order convex optimisation.
method Cutting plane methods, interior point methods, continuous exponential weights, gradient descent, online Newton step.
result Improved existing bounds and algorithms.
Bayesian optimisation tackles high-dimensional categorical and mixed search spaces.
problem Bayesian optimisation on high-dimensional categorical and mixed search spaces is challenging.
method Combining local optimisation with a tailored kernel design.
result Empirically outperforms current baselines in performance and computational costs.
O3 optimizes samples from generative models without extra training.
problem Optimizing specific criteria within rich data distributions.
method Surrogate latent spaces for efficient black-box optimisation.
result Surrogate-space optimisation finds higher-scoring samples.
Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones. Here we present a qu…
A new optimisation method efficiently scales Hessian-vector products for neural networks.
problem Challenges in applying second-order quasi-Newton methods due to large Hessian and non-convexity.
method Proposes an optimisation algorithm that asymptotically uses the exact inverse Hessian with modified eigenvalues.
result Demonstrates scalability and comparable performance to other optimisation methods in neural networks.
Bayesian optimisation framework for graph functions.
problem Optimizing functions on graph structures efficiently.
method Learning suitable kernels and local modelling.
result Demonstrated effectiveness on synthetic and real-world graphs.
Bayesian optimisation tackles expensive black-box functions with constraints.
problem Optimizing constrained black-box functions in machine learning and simulation.
method Proposes a new Knowledge Gradient acquisition function for constrained Bayesian optimisation.
result Demonstrates superior performance over four state-of-the-art constrained Bayesian optimisation algorithms.
MTL2L learns to adapt optimisation rules for unseen data.
problem Learners need to adapt to unseen data domains.
method Introduces MTL2L, a context-aware neural optimiser.
result MTL2L can adapt optimisation rules for unseen data.