No-regret BO algorithm adapts hyperparameters to optimize unknown functions.
problem Misspecification of hyperparameters in BO leads to poor local optima.
method Adapts hyperparameters online to expand function class and converge to optimum.
result First provably no-regret BO algorithm with unknown hyperparameters.
New method optimizes unknown functions adaptively.
problem Tuning hyperparameters in machine learning.
method Gradually and adaptively optimizing an unknown function using estimated gradients.
result Advantages for tuning high dimensional hyperparameters in machine learning.
New bounds quantify estimation error in kernel-based system identification with unknown hyperparameters.
problem Inaccurate error bounds for kernel-based system identification with unknown hyperparameters.
method Construct a high-probability set for true hyperparameters from marginal likelihood, then find worst-case posterior covariance.
result Proposed bounds contain true model with high probability and verified in simulations.
New BO method optimizes functions efficiently even with unknown hyperparameters.
problem Inaccurate estimation of Gaussian process hyperparameters degrades BO performance.
method Exploits multi-armed bandit and novel training loss function for consistent hyperparameter estimation.
result Sub-linear convergence to global optimum with unknown hyperparameters.
The paper analyzes the statistical cost of tuning kernel hyperparameters in robust regression.
problem Finding the best interpolant from a class of kernels with unknown hyperparameters under adversarial noise.
method Finite-sample guarantees, subsampling guarantee for linear regression, ε-net argument for discretizing kernel parameterizations.
result Hyperparameter optimization increases sample complexity by just a logarithmic factor, compared to known parameters.
Bayesian models use hyperparameters to indirectly assign priors, and this work shows how these priors can be derived from maximum entropy principles.
problem Understanding the assumptions and dependencies in Bayesian hierarchical models.
method Demonstrates how canonical distributions and maximum entropy principles can be used to derive marginal priors in hierarchical models.
result Marginal priors in hierarchical models derived from maximum entropy principles have different constraints compared to the original priors.
The Gaussian process (GP) is a popular way to specify dependencies between random variables in a probabilistic model. In the Bayesian framework the covariance structure can be specified using unknown hyperparameters. Integrating over these hyperparameters considers different possible explanations for the data when maki…
We consider the problem of recovering block-sparse signals whose structures are unknown \emph{a priori}. Block-sparse signals with nonzero coefficients occurring in clusters arise naturally in many practical scenarios. However, the knowledge of the block structure is usually unavailable in practice. In this paper, we d…
We consider a network scenario in which agents can evaluate each other according to a score graph that models some interactions. The goal is to design a distributed protocol, run by the agents, that allows them to learn their unknown state among a finite set of possible values. We propose a Bayesian framework in which …
OTSL improves structure learning accuracy with out-of-sample and resampling strategies.
problem Determining optimal hyperparameters for structure learning algorithms.
method Out-of-sample Tuning for Structure Learning (OTSL) using resampling strategies.
result Improves graphical accuracy of structure learning algorithms.
Convolutional Neural Network is known as ConvNet have been extensively used in many complex machine learning tasks. However, hyperparameters optimization is one of a crucial step in developing ConvNet architectures, since the accuracy and performance are reliant on the hyperparameters. This multilayered architecture pa…
Paper proposes a new method for efficient hyperparameter optimization.
problem Challenging task of optimizing hyperparameters in machine learning.
method Sequential Uniform Design (SeqUD) strategy for adaptive and efficient exploration of hyperparameter space.
result The proposed SeqUD strategy outperforms existing methods in hyperparameter optimization.
New algorithm reduces regret bounds for Bayesian optimization with unknown hyperparameters.
problem Optimizing black-box functions with unknown hyperparameters, especially length scale.
method Length Scale Balancing (LB) - aggregating multiple surrogate models with varying length scales.
result LB achieves a regret bound only logaritically away from the oracle algorithm.
We propose a new method for blind system identification. Resorting to a Gaussian regression framework, we model the impulse response of the unknown linear system as a realization of a Gaussian process. The structure of the covariance matrix (or kernel) of such a process is given by the stable spline kernel, which has b…
New algorithm reduces online hyperparameter optimization costs.
problem High cost of evaluating validation examples in online HPO.
method Modeling online HPO as a time-varying Bayesian optimization problem, proposing a costly feedback setting.
result Cost-efficient GP-UCB algorithm reaches human expert-level performance.
New method optimizes hyperparameters for randomized algorithms like random feature regression.
problem Optimizing hyperparameters in randomized algorithms is challenging due to their stochastic nature.
method Introduced a random objective function and used ensemble Kalman inversion (EKI) for gradient-free optimization.
result Demonstrated successful optimization of hyperparameters in various randomized algorithms.
CrossTrainer adapts models across diverse datasets using loss reweighting.
problem Training models with data from varying sources or of varying quality.
method Loss reweighting with optimizations for hyperparameter tuning.
result High model accuracy across various datasets with improved training time.
Prior design is one of the most important problems in both statistics and machine learning. The cross validation (CV) and the widely applicable information criterion (WAIC) are predictive measures of the Bayesian estimation, however, it has been difficult to apply them to find the optimal prior because their mathematic…
Safe Bayesian optimization method using information theory.
problem Optimizing unknown functions while respecting safety constraints.
method Information-theoretic exploration criterion for continuous domains.
result The method learns the value of the safe optimum up to arbitrary precision.
New reshaping method improves hyperparameter search over random search.
problem Improving hyperparameter search efficiency.
method Introducing reshaping techniques to optimize search distributions.
result Substantial gains over random search in various experiments.
An exciting branch of machine learning research focuses on methods for learning, optimizing, and integrating unknown functions that are difficult or costly to evaluate. A popular Bayesian approach to this problem uses a Gaussian process (GP) to construct a posterior distribution over the function of interest given a se…
GANs perform similarly across different synthetic data distributions.
problem Measuring GAN performance on unknown feature spaces.
method Evaluate GANs on explicitly parameterized synthetic data distributions.
result GANs exhibit similar performance trends across different dimensionalities and training set sizes.
New method for adaptive estimation and inference in econometric models without knowing smoothness.
problem Adaptive estimation and inference in ill-posed linear inverse problems with unknown smoothness.
method Discrepancy principle-based framework for adaptive hyperparameter selection.
result Achieves optimal rates in weak and strong metrics for linear functionals.
Improved Kalman filtering with hierarchical variational approach.
problem Inconsistent process covariance estimation and slow convergence speed in traditional variational Kalman filtering.
method Introducing a surrogate variable for process-noise-free state, reformulating CAVI, and sliding-window hyperparameter estimation.
result Enhanced convergence speed and superior estimation accuracy compared to existing methods.
BDC uses Distance Correlation for efficient Bayesian optimization of expensive functions.
problem Efficiently optimizing expensive black-box functions with Bayesian methods.
method Integrates Bayesian optimization with Distance Correlation for automatic exploration and exploitation.
result BDC performs similarly to popular BO methods on benchmark tests and real terrain optimization.
Proposes a model encoder to recommend deep learning architectures for unknown datasets.
problem Choosing an appropriate deep learning architecture for unknown datasets is time-consuming and laborious.
method Proposes a model encoder approach to learn fixed-length representations of architectures and hyperparameters in an unsupervised manner.
result Predicted accuracy of recommended architectures is a good estimator of actual accuracy on unknown datasets.
New method predicts and optimizes matrix recovery from noisy measurements.
problem Recovering rank-1 matrices from Gaussian measurements with noise.
method Stochastic prox-linear iterative algorithm with trajectory predictions.
result The method converges linearly with accurate predictions of error.
We consider the problem of estimating the transition rate matrix of a continuous-time Markov chain from a finite-duration realisation of this process. We approach this problem in an imprecise probabilistic framework, using a set of prior distributions on the unknown transition rate matrix. The resulting estimator is a …
A new method for constrained Bayesian optimization using Max-Value Entropy Search.
problem Optimizing expensive functions with unknown constraints.
method Constrained Max-value Entropy Search (cMES), a novel acquisition function.
result cMES outperforms prior work on constrained hyperparameter optimization problems.
This paper addresses the problem of identifying a lower dimensional space where observed data can be sparsely represented. This under-complete dictionary learning task can be formulated as a blind separation problem of sparse sources linearly mixed with an unknown orthogonal mixing matrix. This issue is formulated in a…
A new method for accurately reconstructing signals without knowing the kernel or signal regularity.
problem Recovering signals from noisy measurements without prior knowledge of the convolution kernel or signal regularity.
method Parametrizing the convolution kernel and prior length-scales, jointly estimated in the inversion procedure.
result Accurate reconstructions of signals with varying regularity and unknown kernel size.
Paper presents a method to reduce prediction variance of DNNs for unknown systems.
problem Uncertainty in DNN predictions due to high variance.
method Ensemble averaging of multiple DNN models trained independently.
result Reduction in variance of DNN predictions, improving reliability.
Sparse linear (or generalized linear) models combine a standard likelihood function with a sparse prior on the unknown coefficients. These priors can conveniently be expressed as a maximization over zero-mean Gaussians with different variance hyperparameters. Standard MAP estimation (Type I) involves maximizing over bo…
New method optimizes costly functions with unknown costs and budget constraints.
problem Optimizing functions with unknown and heterogeneous evaluation costs under a budget constraint.
method Budgeted multi-step expected improvement acquisition function.
result Our method outperforms existing approaches in various synthetic and real problems.
Two algorithms improve GP bandits by selecting priors and minimizing regret.
problem Selecting appropriate GP priors for unknown functions.
method Developed two algorithms: Prior-Elimination GP-TS and HyperPrior GP-TS.
result Established sublinear regret bound for HyperPrior GP-TS.
Self-Tuning Networks optimize hyperparameters using bilevel optimization and gated best-response functions.
problem Optimizing hyperparameters for neural networks.
method Bilevel optimization with gated best-response functions to adapt regularization hyperparameters online.
result Self-Tuning Networks outperform fixed hyperparameter values on large-scale deep learning problems.
We analyze the complexity of Gibbs samplers for inference in crossed random effect models used in modern analysis of variance. We demonstrate that for certain designs the plain vanilla Gibbs sampler is not scalable, in the sense that its complexity is worse than proportional to the number of parameters and data. We thu…
Bayesian optimization finds best hyperparameters for deep learning models.
problem Finding optimal hyperparameters in high-dimensional space.
method Bayesian optimization, Gaussian processes, and adaptive experimentation.
result Ax, BoTorch, and GPyTorch provide a powerful and simple framework for hyperparameter optimization.
A new method uses reinforcement learning for hyperparameter optimization.
problem Optimizing hyperparameters in machine learning models.
method Modeling hyperparameter optimization as a sequential decision problem and using reinforcement learning.
result The method outperforms state-of-the-art approaches for hyperparameter learning.
Extends hyperparameter transfer across model sizes and modules, improving training speed.
problem Training stability and performance of large-scale models with optimal hyperparameters.
method Complete(d) Parameterisation, per-module hyperparameter optimisation and transfer. result Hyperparameter transfer holds even in the per-module hyperparameter regime, improving training speed.
Neural responses are highly variable, and some portion of this variability arises from fluctuations in modulatory factors that alter their gain, such as adaptation, attention, arousal, expected or actual reward, emotion, and local metabolic resource availability. Regardless of their origin, fluctuations in these signal…
HASSO improves SO algorithms by dynamically tuning hyperparameters.
problem Inefficiency of hyperparameter tuning for SO algorithms.
method HASSO is a self-adjusting SO algorithm that dynamically tunes its own hyperparameters.
result HASSO enhances the performance of various SO algorithms across different test problems.
Study shows leaving some hyperparameters at default can outperform tuning them.
problem Importance of tuning hyperparameters in machine learning performance.
method Non-inferiority test and tuning risk methodology.
result Leaving some hyperparameters at default can outperform tuning them.
New algorithm reduces hyperparameter search space using group sparsity.
problem Efficient hyperparameter selection in machine learning.
method Modifies Harmonica algorithm with group-sparse recovery and HyperBand.
result Improves over existing methods like Successive Halving and Random Search.
New method for tuning Graphical Lasso hyperparameters.
problem Tuning hyperparameters of Graphical Lasso.
method Bilevel optimization with first-order method.
result Derivation of Graphical Lasso Jacobian.
PHS optimizes hyperparameters in parallel for expensive computations.
problem Optimizing hyperparameters in computationally expensive tasks.
method Bayesian optimization for parallel hyperparameter search.
result Efficient hyperparameter optimization on multiple instances.
Hyperparameters are critical in machine learning, as different hyperparameters often result in models with significantly different performance. Hyperparameters may be deemed confidential because of their commercial value and the confidentiality of the proprietary algorithms that the learner uses to learn them. In this …
New optimizer improves privacy-protected hyperparameter tuning.
problem No practical methods for differentially private hyperparameter selection.
method Study honest hyperparameter selection under DP, show adaptive optimizers like DPAdam have an advantage.
result DPAdam optimizes hyperparameters more efficiently under DP constraints.