The paper analyzes the statistical cost of tuning kernel hyperparameters in robust regression.
problem Finding the best interpolant from a class of kernels with unknown hyperparameters under adversarial noise.
method Finite-sample guarantees, subsampling guarantee for linear regression, ε-net argument for discretizing kernel parameterizations.
result Hyperparameter optimization increases sample complexity by just a logarithmic factor, compared to known parameters.
Conditional kernel mean embeddings are nonparametric models that encode conditional expectations in a reproducing kernel Hilbert space. While they provide a flexible and powerful framework for probabilistic inference, their performance is highly dependent on the choice of kernel and regularization hyperparameters. Neve…
New bounds quantify estimation error in kernel-based system identification with unknown hyperparameters.
problem Inaccurate error bounds for kernel-based system identification with unknown hyperparameters.
method Construct a high-probability set for true hyperparameters from marginal likelihood, then find worst-case posterior covariance.
result Proposed bounds contain true model with high probability and verified in simulations.
Paper studies kernel hyperparameters for clustering, proposing an efficient search method.
problem Challenges in tuning kernel parameters for clustering, especially for RBF kernels.
method Derives a lower bound for RBF kernel parameters, proposes an efficient hyperparameter search algorithm.
result Proposes an efficient algorithm for hyperparameter search in kernel clustering, improving upon grid search.
This paper tackles hyperparameter tuning for large-scale kernel ridge regression.
problem Hyperparameter tuning is crucial but often left to users, hindering efficiency and usability.
method Proposes a complexity regularization criterion based on a data-dependent penalty for efficient optimization.
result Demonstrates the benefit of the proposed approach through extensive empirical evaluation.
Optimizes Gaussian process hyperparameters using Bayesian autoregression.
problem Optimizing hyperparameters for Matérn kernel temporal Gaussian processes.
method Recursive Bayesian estimation for autoregressive parameters.
result Outperforms traditional optimization methods in runtime and accuracy.
New findings show a balance between data fit and complexity in kernel hyperparameters.
problem Overcorrelation due to reparametrization of kernel hyperparameters.
method Reparametrization of kernel hyperparameters and analysis of marginal likelihood.
result Data fit term influences all other kernel hyperparameters, not just the complexity penalty.
OKSVM optimizes RBF kernel hyperparameter for SVMs, improving classification performance.
problem Intrinsic dependence of RBF kernel hyperparameter on SVM performance.
method Gradient descent method for automatic hyperparameter learning and SVM weights adjustment.
result OKSVM outperforms classical SVM regardless of initial RBF hyperparameter values.
New method optimizes hyperparameters in deep learning models efficiently.
problem Manual hyperparameter tuning in deep learning models is inefficient and requires expertise.
method Introduces lower bounds to the linearized Laplace approximation of the marginal likelihood using neural tangent kernels.
result Optimization of hyperparameters can be significantly accelerated using the method.
Efficiently optimizes hyperparameters for PDE and inverse problems using Gaussian processes.
problem Hyperparameter optimization for scientific computing and inference methods.
method Bilevel optimization with Gauss-Newton linearization for efficient hyperparameter updates.
result Significant improvements in accuracy and robustness compared to random initialization.
The hyperparameters in Gaussian process regression (GPR) model with a specified kernel are often estimated from the data via the maximum marginal likelihood. Due to the non-convexity of marginal likelihood with respect to the hyperparameters, the optimization may not converge to the global maxima. A common approach to …
Bayesian optimisation improves with fully-Bayesian treatment of hyperparameters.
problem Overconfident model predictions in BO due to ignoring hyperparameter uncertainty.
method Investigate FBBO using three approximate inference schemes compared to maximum likelihood approach.
result FBBO using EI with an ARD kernel leads to best performance in noise-free setting.
Derives a family of hyperparameter scaling strategies for neural networks.
problem Optimizing hyperparameters for wide and deep neural networks.
method Introduces a one-parameter family of hyperparameter scaling strategies.
result Reveals proper scaling of depth with width for large-scale models.
BOOST automates kernel and acquisition function selection in Bayesian optimization.
problem Inappropriate kernel and acquisition function combinations lead to poor performance in Bayesian optimization.
method BOOST uses offline evaluation to predict and select the best kernel-acquisition function pair.
result BOOST consistently improves over fixed-hyperparameter BO and is competitive with state-of-the-art adaptive methods.
Deep Gaussian Processes with polynomial kernels can collapse rapidly without proper hyperparameter tuning.
problem The collapse of Deep Gaussian Processes with polynomial kernels without careful hyperparameter tuning.
method Analysis using the Berry-Esseen Theorem and observation of prior behavior.
result The prior of a Deep Gaussian Process collapses rapidly towards zero or places negligible mass on low norm functions without proper hyperparameter tuning.
New algorithm improves Gaussian process hyperparameter tuning for large datasets.
problem Scalable hyperparameter tuning for Gaussian processes on large datasets.
method Estimates smoothness and length-scale parameters in Matern kernel using novel loss functions.
result Improved uncertainty quantification over traditional methods.
New random feature maps for Laplacian and related kernels.
problem Challenges in approximating the Laplacian kernel and its generalizations.
method Developed random feature maps for Laplacian and related kernels, providing efficient sampling schemes.
result Demonstrated the efficacy of these random feature maps on real datasets.
Computational efficiency is an important consideration for deploying machine learning models for time series prediction in an online setting. Machine learning algorithms adjust model parameters automatically based on the data, but often require users to set additional parameters, known as hyperparameters. Hyperparamete…
Boosting combines weak (biased) learners to obtain effective learning algorithms for classification and prediction. In this paper, we show a connection between boosting and kernel-based methods, highlighting both theoretical and practical applications. In the context of ℓ2 boosting, we start with a weak linear le…
Paper proves convergence rates for Gaussian kernel ridge regression.
problem Understanding convergence rates for Gaussian kernel ridge regression.
method Establishes polynomial convergence rates for KRR with fixed hyperparameters.
result First polynomial convergence rates for Gaussian kernel ridge regression.
This paper uses Nested Sampling to improve Gaussian Process uncertainty quantification.
problem Underestimating predictive uncertainty and overfitting in Gaussian Process models.
method Marginalises hyperparameters using Nested Sampling for spectral mixture kernels.
result Improves predictive performance and uncertainty quantification across various data sets.
Proposes LAMA for principled SVDD hyperparameter estimation.
problem Estimating SVDD hyperparameters is difficult and heuristic methods are unreliable.
method LAMA (Local Active Min-Max Alignment) using active learning and kernel alignment.
result LAMA provides evidence-based estimates for SVDD hyperparameters with quality scores.
Conditional kernel mean embeddings form an attractive nonparametric framework for representing conditional means of functions, describing the observation processes for many complex models. However, the recovery of the original underlying function of interest whose conditional mean was observed is a challenging inferenc…
Bayesian optimization (BO) based on Gaussian process models is a powerful paradigm to optimize black-box functions that are expensive to evaluate. While several BO algorithms provably converge to the global optimum of the unknown function, they assume that the hyperparameters of the kernel are known in advance. This is…
We propose Bayesian extensions of two nonparametric regression methods which are kernel and mutual k-nearest neighbor regression methods. Derived based on Gaussian process models for regression, the extensions provide distributions for target value estimates and the framework to select the hyperparameters. It is show…
Sparse Gaussian process hyperparameters optimized using MCMC.
problem Hyperparameter uncertainty leads to biased estimates and underestimation of predictive uncertainty.
method Proposes an MCMC algorithm to sample from the hyperparameter posterior in sparse Gaussian process regression.
result Significantly improves sampling efficiency in the Gaussian likelihood case.
A new method for kernel tests without data splitting increases power.
problem Lack of power in kernel-based tests due to data splitting.
method Selective inference framework to learn hyperparameters and test on full sample.
result Empirically larger test power without data splitting, regardless of split proportion.
Proposes a new feature preprocessing method using kernel density integral transformation.
problem Feature preprocessing for tabular data in machine learning and statistics.
method Kernel density integral transformation as a drop-in replacement or improved alternative to min-max scaling and quantile transformation.
result Frequently outperforms min-max scaling and quantile transformation with hyperparameter tuning.
Empirical comparison of 18 hyperparameter tuning algorithms for SVM.
problem Tuning hyperparameters C and γ for SVM with RBF kernel. method Compared 18 search algorithms on 115 real-life data sets.
result Trees of Parzen estimators and particle swarm optimization perform similarly to grid search.
New tools evaluate and optimize conditional sequence models in bioinformatics.
problem Evaluating and optimizing conditional sequence models in bioinformatics.
method Kernel-based discrepancy measure (ACMMD) to estimate model fit and tune hyperparameters.
result Rejects the hypothesis that ProteinMPNN fits its data for various protein families and optimizes model temperature.
A two-stage GPR framework with automatic kernel search and subsampling improves prediction accuracy and efficiency.
problem Inaccurate predictions due to misspecified mean and kernel functions in Gaussian Process Regression.
method Two-stage GPR, automatic kernel search, subsampling for hyperparameter initialization.
result Competitive or better performance compared to full dataset training, robust on real-world datasets.
We propose a new method for blind system identification. Resorting to a Gaussian regression framework, we model the impulse response of the unknown linear system as a realization of a Gaussian process. The structure of the covariance matrix (or kernel) of such a process is given by the stable spline kernel, which has b…
Kernel methods are one of the mainstays of machine learning, but the problem of kernel learning remains challenging, with only a few heuristics and very little theory. This is of particular importance in methods based on estimation of kernel mean embeddings of probability measures. For characteristic kernels, which inc…
GP model for time series forecasting with priors.
problem Automatic selection of optimal kernels and reliable estimation of hyperparameters.
method Fixed composition of kernels, automatic relevance determination (ARD), empirical Bayes priors.
result GP model is more accurate than state-of-the-art models.
In likelihood-free settings where likelihood evaluations are intractable, approximate Bayesian computation (ABC) addresses the formidable inference task to discover plausible parameters of simulation programs that explain the observations. However, they demand large quantities of simulation calls. Critically, hyperpara…
Bayesian Optimization (BO) has become a core method for solving expensive black-box optimization problems. While much research focussed on the choice of the acquisition function, we focus on online length-scale adaption and the choice of kernel function. Instead of choosing hyperparameters in view of maximum likelihood…
Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using in…
Bayesian optimisation is a popular technique for hyperparameter learning but typically requires initial exploration even in cases where similar prior tasks have been solved. We propose to transfer information across tasks using learnt representations of training datasets used in those tasks. This results in a joint Gau…
In this paper, we propose the use of a black-box optimization method called deterministic Mesh Adaptive Direct Search (MADS) algorithm with orthogonal directions (Ortho-MADS) for the selection of hyperparameters of Support Vector Machines with a Gaussian kernel. Different from most of the methods in the literature that…
Optimal kernel in KR can be data-dependent, improving model performance.
problem Fixed kernel in KR limits model performance.
method Considered data-dependent kernels for KR, using posterior covariance.
result Data-dependent kernel choice leads to optimal performance.
New framework for understanding infinite-width neural networks.
problem Understanding the infinite-width limit behavior of neural networks.
method General framework to study limit behavior of neural models based on hyperparameter scaling.
result Derives scaling for existing mean-field and neural tangent kernel limits and introduces new dynamically stable limits.
Decentralized Gaussian processes for multi-agent systems.
problem Scalable and flexible learning solutions for multi-agent systems.
method Asymptotically exact decentralized solution to Gaussian processes, with online Bayesian model averaging for hyperparameter selection.
result Asymptotically exact decentralized Gaussian process approximation and online Bayesian model averaging.
Improves Bayesian optimization efficiency for mixed variable spaces.
problem Boosting sample efficiency in Bayesian optimization for mixed variable spaces.
method Proposes frequency modulated (FM) kernels to model complex dependencies across different types of variables.
result BO-FM outperforms competitors in various optimization problems.
Proposes FMS for more efficient neural network hyperparameter optimization.
problem Efficient hyperparameter optimization for deep learning models.
method Uses logged checkpoints of trained weights to guide hyperparameter selections.
result Proposes Forecasting Model Search (FMS) method.
New recommendations improve Gaussian process accuracy and stability.
problem Numerical instabilities and poor test likelihoods in iterative Gaussian process learning.
method Investigated CG tolerance, preconditioner rank, and Lanczos decomposition rank. Recommended small CG tolerance and large root decomposition size.
result L-BFGS-B optimizer achieves convergence with fewer gradient updates, improving Gaussian process accuracy.
Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter op…
New algorithm selects multiple kernels for better GP regression predictions.
problem Improving Gaussian process regression accuracy with multiple kernels.
method Variational Bayesian kernel selection (VBKS) for sparse Gaussian process regression (SGPR).
result VBKS learns uncertainty in kernel selection for better predictions.
Estimates KRR risk from training data for various kernels and hyperparameters.
problem Predicting the generalization error of Kernel Ridge Regression.
method Introduces SCT and KARE to approximate KRR risk from training data.
result KARE provides an excellent approximation of KRR risk and helps select good kernels.