Novel hyperparameter optimization for target tasks under covariate shift.
problem Hyperparameter optimization under multi-source covariate shift.
method Construct variance reduced estimator to unbiasedly approximate target objective; propose no-regret hyperparameter optimization procedure.
result Proposed framework broadens applications of automated hyperparameter optimization.
The Gaussian process (GP) is a popular way to specify dependencies between random variables in a probabilistic model. In the Bayesian framework the covariance structure can be specified using unknown hyperparameters. Integrating over these hyperparameters considers different possible explanations for the data when maki…
Improved Kalman filtering with hierarchical variational approach.
problem Inconsistent process covariance estimation and slow convergence speed in traditional variational Kalman filtering.
method Introducing a surrogate variable for process-noise-free state, reformulating CAVI, and sliding-window hyperparameter estimation.
result Enhanced convergence speed and superior estimation accuracy compared to existing methods.
The correlation length-scale next to the noise variance are the most used hyperparameters for the Gaussian processes. Typically, stationary covariance functions are used, which are only dependent on the distances between input points and thus invariant to the translations in the input space. The optimization of the hyp…
New bounds quantify estimation error in kernel-based system identification with unknown hyperparameters.
problem Inaccurate error bounds for kernel-based system identification with unknown hyperparameters.
method Construct a high-probability set for true hyperparameters from marginal likelihood, then find worst-case posterior covariance.
result Proposed bounds contain true model with high probability and verified in simulations.
Method estimates noise variance in Gaussian process regression.
problem Estimating noise variance in Gaussian process regression models.
method Reduces hyperparameter space, uses marginal likelihood function, derives bounds and asymptotes.
result Computational advantages and robustness compared to traditional methods.
SPICE estimates sparse linear dynamic networks without hyperparameters.
problem Estimating topology and dynamics of sparse linear dynamic networks.
method SPICE (Sparse Iterative Covariance Estimation) method in an iterative framework.
result Directly reveals the underlying topology of the network.
MuyGPs efficiently estimates GP hyperparameters using local cross-validation.
problem Efficiently estimating GP hyperparameters for large datasets.
method Uses nearest neighbors structure and leave-one-out cross-validation.
result Outperforms state-of-the-art competitors in time and prediction accuracy.
Develops a new MCMC-based Wishart prior for Gaussian Process covariance matrix.
problem Difficult inference for multivariate Gaussian Processes with multiple lengthscale parameters.
method Introduces a self-assembled Wishart prior and uses MCMC for Bayesian inference on kernel hyperparameters.
result Demonstrates the effectiveness of the new prior in GP-based learning with empirical results.
Optimizes Gaussian process hyperparameters using Bayesian autoregression.
problem Optimizing hyperparameters for Matérn kernel temporal Gaussian processes.
method Recursive Bayesian estimation for autoregressive parameters.
result Outperforms traditional optimization methods in runtime and accuracy.
This study improves hyperparameter optimization for categorical and non-normal data.
problem Bayesian hyperparameter optimization struggles with categorical hyperparameters and non-normal data.
method Integrates conformalized quantile regression to address estimation weaknesses and provides robust calibration guarantees.
result Quantile surrogate architectures and acquisition functions yield superior performance compared to existing methods.
This paper explores approximations for fully Bayesian Gaussian Process Regression.
problem Learning in Gaussian Process models through hyperparameter adaptation.
method Two approximation schemes: Hamiltonian Monte Carlo and Variational Inference.
result Predictive performance analysis on various benchmark datasets.
New algorithm improves Gaussian process hyperparameter tuning for large datasets.
problem Scalable hyperparameter tuning for Gaussian processes on large datasets.
method Estimates smoothness and length-scale parameters in Matern kernel using novel loss functions.
result Improved uncertainty quantification over traditional methods.
Federated learning method improves covariate shift adaptation for missing target values.
problem Missing target values in federated learning.
method Federated covariate shift adaptation algorithm for missing target output values.
result Asymptotically unbiased and efficient algorithm for federated learning.
This paper presents a Bayesian image segmentation model based on Potts prior and loopy belief propagation. The proposed Bayesian model involves several terms, including the pairwise interactions of Potts models, and the average vectors and covariant matrices of Gauss distributions in color image modeling. These terms a…
Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like G…
Novel probabilistic solver speeds up solving related linear systems.
problem Efficiently solving multiple related linear systems.
method Probabilistic linear solver over the parameter space, leveraging solved systems.
result Faster and more efficient solution of related linear systems.
Proposes a method to optimize neural network initialization using marginal likelihood maximization.
problem Optimizing hyperparameters for neural network initialization.
method Leverages the connection between neural networks and Gaussian processes to infer optimal hyperparameters.
result Marginal likelihood maximization provides near-optimal prediction performance on MNIST classification tasks.
In this work, we propose a new Gaussian process regression (GPR) method: physics information aided Kriging (PhIK). In the standard data-driven Kriging, the unknown function of interest is usually treated as a Gaussian process with assumed stationary covariance with hyperparameters estimated from data. In PhIK, we compu…
The paper presents a novel approach to direct covariance function learning for Bayesian optimisation, with particular emphasis on experimental design problems where an existing corpus of condensed knowledge is present. The method presented borrows techniques from reproducing kernel Banach space theory (specifically m-k…
We speed up Gaussian process cross-validation calculations and improve model diagnostics.
problem Efficiently calculating cross-validation residuals and their covariances in Gaussian processes.
method Generalized fast Gaussian process leave-one-out formulae to multiple-fold cross-validation, highlighting covariance structures.
result Correcting for residual covariances in cross-validation improves back to Maximum Likelihood Estimation.
This paper proposes a novel scheme for reduced-rank Gaussian process regression. The method is based on an approximate series expansion of the covariance function in terms of an eigenfunction expansion of the Laplace operator in a compact subset of Rd. On this approximate eigenbasis the eigenvalues of the c…
Flexible co-data learning improves clinical prediction models.
problem High-dimensional clinical data challenges prediction accuracy.
method Combining domain knowledge and external studies to estimate adaptive multi-group ridge penalties.
result Improves prediction performance and variable selection stability.
The paper proposes a method to learn hyperparameters without validation sets, improving efficiency and accuracy.
problem Training large models on limited data to avoid overfitting and reduce validation set usage.
method Gradient-based learning of hyperparameters via a data-emphasized evidence lower bound (ELBO) objective.
result The data-emphasized ELBO reduces hyperparameter search time from 88+ hours to under 3 hours while maintaining comparable accuracy.
The paper shows cross-validation fails in learning Gaussian graphical model structures.
problem Cross-validation's failure in learning Gaussian graphical model structures.
method Finite-sample bounds on misidentification probability of Lasso estimator.
result Cross-validation is inconsistent for learning Gaussian graphical model structures.
The Poisson model is frequently employed to describe count data, but in a Bayesian context it leads to an analytically intractable posterior probability distribution. In this work, we analyze a variational Gaussian approximation to the posterior distribution arising from the Poisson model with a Gaussian prior. This is…
Paper develops methods for estimating GLMs and SNR under proportional asymptotics.
problem Estimation of regression coefficients and SNR in high-dimensional GLMs.
method Method-of-Moments type estimators that bypass nuisance function estimation.
result Consistent and asymptotically normal estimators derived for targets of inference.
In the analysis of sequential data, the detection of abrupt changes is important in predicting future changes. In this paper, we propose statistical hypothesis tests for detecting covariance structure changes in locally smooth time series modeled by Gaussian Processes (GPs). We provide theoretically justified threshold…
We propose a new method for blind system identification. Resorting to a Gaussian regression framework, we model the impulse response of the unknown linear system as a realization of a Gaussian process. The structure of the covariance matrix (or kernel) of such a process is given by the stable spline kernel, which has b…
Dynamic paired comparison models, such as Elo and Glicko, are frequently used for sports prediction and ranking players or teams. We present an alternative dynamic paired comparison model which uses a Gaussian Process (GP) as a prior for the time dynamics rather than the Markovian dynamics usually assumed. In addition,…
A new approach simplifies multitask Gaussian processes without rank approximations.
problem Handling multioutput regression problems with conditionally dependent tasks.
method Introduces a novel approach to reduce multitask learning to univariate GPs, eliminating the need for rank approximations.
result Accurately recovers multitask covariance and noise matrices with fewer parameters, improving performance and reducing overfitting risk.
Proposes using external data to improve predictions in medical applications with limited samples.
problem Small sample sizes and complex covariate-response relationships in medical data.
method Integrates external co-data into Bayesian Additive Regression Trees (BART) using an empirical Bayes framework.
result Improves prediction accuracy compared to standard BART, especially for nonlinear relationships.
This work improves texture segmentation by automatically tuning hyperparameters for Total-Variation.
problem The challenge is to automatically select hyperparameters for Total-Variation texture segmentation.
method The approach involves extending Stein's unbiased gradient estimator to handle correlated Gaussian noise, leading to an automatic tuning method.
result The method provides an automatic way to select hyperparameters for Total-Variation texture segmentation.
LLMs struggle to optimize hyperparameters efficiently, but hybrid methods can improve performance.
problem Optimizing hyperparameters of small language models using LLMs.
method Comparison of classical HPO algorithms and LLM-based methods, introducing Centaur hybrid approach.
result Hybrid Centaur approach achieves best results, outperforming classical and pure LLM methods.
GANs mode collapse solved with Bures distance.
problem GANs mode collapse or mode dropping.
method Use Bures distance to match real and fake batch diversity in feature space.
result Diversity matching reduces mode collapse and improves sample quality.
New GMM models fit high-dimensional data with fewer parameters.
problem Overparameterization and lack of flexibility in GMMs for high-dimensional data.
method Piecewise-constant covariance eigenvalue profiles, EM and penalized EM algorithms.
result Superior likelihood-parsimony tradeoffs in density fitting, clustering, and denoising.
Gaussian variational approximation is a popular methodology to approximate posterior distributions in Bayesian inference especially in high dimensional and large data settings. To control the computational cost while being able to capture the correlations among the variables, the low rank plus diagonal structure was in…
Solution to sparse PCA tuning problem using Empirical Bayes.
problem Sparse PCA multiple tuning problem (MTP).
method Empirical Bayes covariance decomposition for penalized PCA.
result Empirical Bayes approach efficiently solves MTP in sparse PCA.
Current adoption of machine learning in industrial, societal and economical activities has raised concerns about the fairness, equity and ethics of automated decisions. Predictive models are often developed using biased datasets and thus retain or even exacerbate biases in their decisions and recommendations. Removing …
Study compares different covariance estimation methods for portfolio allocation.
problem Comparing methods for estimating covariance and precision matrices in portfolio allocation.
method Gaussian Graphical Model (GGM), Shrinkage, Thresholding, Random Matrix Theory (RMT) methods.
result GGM methods outperform other methods in predictive ability for portfolio allocation.
Spectral mixture (SM) kernels comprise a powerful class of generalized kernels for Gaussian processes (GPs) to describe complex patterns. This paper introduces model compression and time- and phase (TP) modulated dependency structures to the original (SM) kernel for improved generalization of GPs. Specifically, by adop…
The problem of combined state and input estimation of linear structural systems based on measured responses and a priori knowledge of structural model is considered. A novel methodology using Gaussian process latent force models is proposed to tackle the problem in a stochastic setting. Gaussian process latent force mo…
Gaussian Processes offer a flexible method for modeling and predicting outcomes with uncertainty estimates.
problem Capturing uncertainty in predictions at new data points, especially with poor overlap and extrapolation.
method Gaussian Processes model a posterior distribution over outcomes, reflecting the range of plausible models.
result GPs provide a principled approach to handling extrapolation and uncertainty in predictions.
A two-stage GPR framework with automatic kernel search and subsampling improves prediction accuracy and efficiency.
problem Inaccurate predictions due to misspecified mean and kernel functions in Gaussian Process Regression.
method Two-stage GPR, automatic kernel search, subsampling for hyperparameter initialization.
result Competitive or better performance compared to full dataset training, robust on real-world datasets.
The paper uses random matrix theory for multi-task regression, improving time series forecasting.
problem Improving time series forecasting using multi-task regression.
method Applying random matrix theory to multi-task regression problems, deriving closed-form solutions for optimization.
result Provides a robust foundation for hyperparameter optimization in multi-task regression scenarios.
A new IPM uses ReLU networks to measure probability discrepancies.
problem Measuring the difference between two probability distributions in high dimensions.
method Proposes a new parametric IPM using ReLU neural networks to optimize and distinguish between distributions.
result The proposed IPM has good convergence rates and can be used as a surrogate for other IPMs.
Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters …
We introduce the convolutional spectral kernel (CSK), a novel family of non-stationary, nonparametric covariance kernels for Gaussian process (GP) models, derived from the convolution between two imaginary radial basis functions. We present a principled framework to interpret CSK, as well as other deep probabilistic mo…