arXiv research

One-pass optimisation for high-dimensional hyperparameters.

A new optimisation framework for neural networks without hyperparameters.

Extends hyperparameter transfer across model sizes and modules, improving training speed.

Optimizes one-class classification methods for better performance.

Bayesian optimisation improves with fully-Bayesian treatment of hyperparameters.

The paper speeds up hyperparameter optimisation in Gaussian processes.

Regularisation improves ML classifier stability against poisoning attacks.

The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computationally expensive. More advanced methods like Population Based Training that learn optimal schedule…

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for whic…

AdamZ optimiser improves neural network training efficiency.

Aims to describe neural network training dynamics using two-time-scale models.

Automated Budget Constrained Training optimizes model training under time constraints.

Bayesian optimisation tackles expensive black-box functions with constraints.

A new method learns priors for Bayesian optimisation to improve performance.

Revisits online Laplace methods for neural networks, showing they are sound under certain conditions.

Batch Bayesian optimisation (BO) has been successfully applied to hyperparameter tuning using parallel computing, but it is wasteful of resources: workers that complete jobs ahead of others are left idle. We address this problem by developing an approach, Penalising Locally for Asynchronous Bayesian Optimisation on $k$…

Bayesian optimisation is a popular technique for hyperparameter learning but typically requires initial exploration even in cases where similar prior tasks have been solved. We propose to transfer information across tasks using learnt representations of training datasets used in those tasks. This results in a joint Gau…

Novel probabilistic solver speeds up solving related linear systems.

The notion of expense in Bayesian optimisation generally refers to the uniformly expensive cost of function evaluations over the whole search space. However, in some scenarios, the cost of evaluation for black-box objective functions is non-uniform since different inputs from search space may incur different costs for …

It is typical for a machine learning system to have numerous hyperparameters that affect its learning rate and prediction quality. Finding a good combination of the hyperparameters is, however, a challenging job. This is mainly because evaluation of each combination is extremely expensive computationally; indeed, train…

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are com…

This paper uses Nested Sampling to improve Gaussian Process uncertainty quantification.

Bayesian Optimisation (BO) refers to a suite of techniques for global optimisation of expensive black box functions, which use introspective Bayesian models of the function to efficiently search for the optimum. While BO has been applied successfully in many applications, modern optimisation tasks usher in new challeng…

AEGiS optimizes expensive function evaluations asynchronously.

New algorithm reduces regret bounds for Bayesian optimization with unknown hyperparameters.

New methods improve global optimisation for expensive functions using lookahead strategies.

Bayesian optimisation is improved by incorporating expert prior through space warping.

Improves meta-learning efficiency with mixed-mode differentiation.

Bayesian optimisation (BO) is a well-known efficient algorithm for finding the global optimum of expensive, black-box functions. The current practical BO algorithms have regret bounds ranging from $\mathcal{O}(\frac{logN}{\sqrt{N}})$ to $\mathcal O(e^{-\sqrt{N}})$, where $N$ is the number of evaluations. This paper exp…

Sparse pseudo-point approximations for Gaussian process (GP) models provide a suite of methods that support deployment of GPs in the large data regime and enable analytic intractabilities to be sidestepped. However, the field lacks a principled method to handle streaming data in which both the posterior distribution ov…

New algorithms for sampling in constrained domains without learning rates.

Investigates gradient descent dynamics and introduces new regularisation methods.

Warm starts improve Gaussian process regression by up to 16x.

When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed…

This paper proposes using neural networks for Bayesian optimisation in machine learning.

The paper presents a novel approach to direct covariance function learning for Bayesian optimisation, with particular emphasis on experimental design problems where an existing corpus of condensed knowledge is present. The method presented borrows techniques from reproducing kernel Banach space theory (specifically m-k…

Multi-label classification is an approach which allows a datapoint to be labelled with more than one class at the same time. A common but trivial approach is to train individual binary classifiers per label, but the performance can be improved by considering associations within the labels. Like with any machine learnin…

This paper employs machine learning algorithms to forecast German electricity spot market prices. The forecasts utilize in particular bid and ask order book data from the spot market but also fundamental market data like renewable infeed and expected demand. Appropriate feature extraction for the order book data is dev…

New HPO frameworks needed for continual learning.

This study compares 6 imitation learning algorithms using a common dataset and hyperparameter budget.

Paper uses ensemble learning for IoT cybersecurity anomaly detection.

Deep neural networks continue to show improved performance with increasing depth, an encouraging trend that implies an explosion in the possible permutations of network architectures and hyperparameters for which there is little intuitive guidance. To address this increasing complexity, we propose Evolutionary DEep Net…

New algorithm optimizes noisy function evaluations with delayed feedback.

MVRSM optimizes expensive functions with mixed variables, outperforming state-of-the-art methods.

Method estimates uncertainty in CT reconstructions.

Develops methods to measure and set function-space learning rates in neural networks.

Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies. On-policy methods bring many benefits, such as ability to evaluate each resulting policy. However, they usually d…

AdamQLR optimizes Adam with K-FAC heuristics, achieving comparable performance to tuned benchmarks.