New analysis shows a gap between Gaussian RKHS and neural networks on unbounded domains.
problem Understanding the function space bias of neural networks compared to Gaussian RKHS.
method Infinite-center asymptotic analysis of neural network Banach space and Gaussian RKHS on unbounded domains.
result Certain functions in Gaussian RKHS have infinite norm in neural network Banach space on unbounded domains.
Enhances DGPs with adaptive RKHS Fourier features for better non-stationary pattern modeling.
problem Capturing complex non-stationary patterns in non-linear dynamical systems.
method Integrates ODE-based RKHS Fourier features into DGPs using convolution operations for adaptive amplitude and phase modulation. Uses a doubly stochastic variational inference framework.
result Improved predictive performance across various regression tasks.
Reconstruction of a function from noisy data is often formulated as a regularized optimization problem over an infinite-dimensional reproducing kernel Hilbert space (RKHS). The solution describes the observed data and has a small RKHS norm. When the data fit is measured using a quadratic loss, this estimator has a know…
This study connects Gaussian processes and RKHS, bridging two machine learning communities.
problem Understanding the relationship between Gaussian processes and RKHS.
method Examining connections and equivalences in regression, interpolation, and other topics.
result Established the equivalence between Gaussian Hilbert space and RKHS.
Study convergence and approximations of entropic regularized Wasserstein distances for Gaussian and RKHS measures.
problem Convergence and approximations of entropic regularized Wasserstein distances in Gaussian and RKHS settings.
method Analysis of convergence and finite sample approximations of entropic regularized Wasserstein distances in Gaussian and RKHS settings.
result Strictly weaker convergence in 2-Sinkhorn divergence for Gaussian measures compared to exact 2-Wasserstein distance.
Study proposes a new metric for comparing Gaussian mixtures in RKHS.
problem Comparing complex multimodal densities in RKHS.
method Wasserstein-type metric for kernel Gaussian mixtures.
result Enhanced capability to model multimodal densities.
This study approximates distances between Gaussian processes and covariance operators using RKHS.
problem Approximating distances between Gaussian processes and covariance operators from finite samples.
method Using reproducing kernel Hilbert space (RKHS) covariance and cross-covariance operators, the study shows how to consistently and efficiently estimate Sinkhorn divergence from finite samples.
result Convergence rates are dimension-independent and of the same order as Hilbert-Schmidt distance.
The paper develops divergences for Gaussian processes and RKHS settings.
problem Estimating divergences in infinite-dimensional spaces.
method Formulations of Alpha Log-Det divergences, continuity in norm, laws of large numbers, consistent estimation from finite samples.
result Infinite-dimensional divergences can be estimated from finite-dimensional versions with dimension-independent sample complexities.
Improved GP bandit algorithms for noiseless, varying noise, and RKHS norms.
problem Minimizing regret in Gaussian process bandits with unknown reward functions.
method New upper bound on maximum posterior variance, refined MVR and PE algorithms.
result Optimal regret bounds for noiseless, varying noise, and RKHS norms.
Optimal simple regret bound for Gaussian Process bandits.
problem Sequential optimization of expensive-to-evaluate functions.
method Proved a bound on simple regret for pure exploration algorithms.
result Order optimal bound on simple regret for Gaussian Process bandits.
New framework assesses regularization norms in ill-posed problems, revealing L2 instability and proposing adaptive fractional RKHS solutions.
problem Comparative analysis of regularization norms in ill-posed problems.
method Small noise analysis framework for Tikhonov and RKHS regularizations.
result Optimal convergence rates achieved with adaptive fractional RKHS, but hyper-parameters decay too fast.
The paper identifies a 'small' set of functions containing Gaussian process samples.
problem Identifying a small set of functions containing Gaussian process samples.
method Using scaled RKHSs and Karhunen-Loève theorem, the paper defines the sample support set.
result The sample support set consists of functions with bounded squared basis coefficients.
Neural networks can approximate functionals on RKHS with error bounds.
problem Approximating functionals on RKHS using neural networks.
method Interpolating orthogonal projections in RKHS using point evaluations.
result Explicit error bounds for various kernels (inverse multiquadric, Gaussian, Sobolev).
We generalize the orthonormal basis for the Gaussian RKHS described in \cite{MinhGaussian2010} to an infinite, continuously parametrized, family of orthonormal bases, along with some implications. The proofs are direct generalizations of those in \cite{MinhGaussian2010}.
A scalable algorithm approximates Bayesian posteriors in RKHS with improved efficiency.
problem Scalable inference for Bayes posteriors in infinite-dimensional spaces.
method Approximate Langevin diffusion projection onto first M components, using law of total probability and sufficiency assumption.
result The method recovers SVGP as a special case and is provably close to optimal for convex and Lipschitz continuous likelihoods.
Study on estimating distances between covariance operators and Gaussian processes.
problem Estimating distances between covariance operators and Gaussian processes.
method Riemannian distances, concentration results for Hilbert space-valued random variables, RKHS covariance and cross-covariance operators.
result Both distances converge in the Hilbert-Schmidt norm and can be consistently and efficiently estimated.
Differential privacy is a framework for privately releasing summaries of a database. Previous work has focused mainly on methods for which the output is a finite dimensional vector, or an element of some discrete set. We develop methods for releasing functions while preserving differential privacy. Specifically, we sho…
We propose a representation of Gaussian processes (GPs) based on powers of the integral operator defined by a kernel function, we call these stochastic processes integral Gaussian processes (IGPs). Sample paths from IGPs are functions contained within the reproducing kernel Hilbert space (RKHS) defined by the kernel fu…
New algorithms improve GP inference without approximations, achieving better results.
problem Inexact stochastic optimization methods in Gaussian Processes leading to biased results.
method Exact stochastic inference for GPs with finite dimensional RKHS, extending to infinite dimensions.
result Achieves better experimental results than existing methods in constrained resource settings.
Researchers improve Gaussian processes to model inconsistent preferences.
problem Model inconsistent preferences and clusters of comparable items.
method Generalized Gaussian processes with spectral decomposition and universal RKHS.
result Competitive with state-of-the-art methods on simulated and real-world data.
We propose a generic spatiotemporal event forecasting method, which we developed for the National Institute of Justice's (NIJ) Real-Time Crime Forecasting Challenge. Our method is a spatiotemporal forecasting model combining scalable randomized Reproducing Kernel Hilbert Space (RKHS) methods for approximating Gaussian …
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Study learns a projection and function in Gaussian models.
problem Learning a one-dimensional projection and a univariate function in high-dimensional Gaussian models.
method Gradient flow dynamics of alternating scheme, RKHS adaptation.
result Gradient flow dynamics converge with rate controlled by Gaussian regularity.
The main contribution of the paper is to show that Gaussian sketching of a kernel-Gram matrix K yields an operator whose counterpart in an RKHS H, is a \emph{random projection} operator---in the spirit of Johnson-Lindenstrauss (J-L) lemma. To be precise, given a random matrix Z with i.i.d. Ga…
Paper learns optimal kernels for Gaussian process regression in aerodynamics.
problem Approximating complex functions from limited data in aerodynamics.
method Two algorithms: Kernel Flow and Spectral Kernel Ridge Regression.
result Explicit construction of optimal kernels based on target function features.
Revisits Gaussian process model with spherical harmonics for scalable deep learning.
problem Scaling Gaussian process models to large input dimensions with high frequency learning.
method Introduces new kernels related to deep models, variational learning of spherical harmonic phases, and sparseness in eigenbasis.
result Enables scaling to larger input dimensions and learning of high frequency variations.
New algorithms tackle RKHS bandits with reduced complexity and improved performance.
problem Adversarial and stochastic RKHS bandit problems with high computational complexity.
method Combining approximation theory with misspecified linear bandit methods.
result First general algorithm for adversarial RKHS bandit problem.
Constructs non-asymptotic confidence regions for unknown functions in RKHS.
problem Global probabilistic confidence regions for unknown functions in RKHS.
method Reduces confidence region construction to estimating RKHS norm.
result Valid confidence regions can be constructed non-asymptotically.
New algorithm optimizes Hölder smooth functions in RKHS with tighter regret bounds.
problem Optimizing Hölder smooth functions in RKHS with bounded norm.
method Proposes a new algorithm ( exttt{LP-GP-UCB}) using Local Polynomial (LP) estimators and multi-scale UCB.
result Derives high probability bounds on simple and cumulative regret, matching optimal performance for SE kernel and uniformly tighter bounds for Matérn kernels.
Deep neural kernels and Laplace kernel have equivalent RKHS on spheres.
problem Comparing RKHS of deep neural tangent and Laplace kernels.
method Proof of RKHS equivalence using sphere restrictions and kernel properties.
result RKHS of deep neural tangent kernel and Laplace kernel are the same on Sd−1. Statistical machine learning plays an important role in modern statistics and computer science. One main goal of statistical machine learning is to provide universally consistent algorithms, i.e., the estimator converges in probability or in some stronger sense to the Bayes risk or to the Bayes decision function. Kerne…
Kernel methods are powerful tools to capture nonlinear patterns behind data. They implicitly learn high (even infinite) dimensional nonlinear features in the Reproducing Kernel Hilbert Space (RKHS) while making the computation tractable by leveraging the kernel trick. Classic kernel methods learn a single layer of nonl…
Improves probability distribution compression with KT algorithm.
problem Efficiently compressing probability distributions.
method Kernel thinning (KT) algorithm with four improvements.
result KT yields tighter, dimension-free guarantees for any kernel.
This paper develops a general framework for metric learning in RKHS with theoretical guarantees.
problem Learning a metric in RKHS from triplet comparisons.
method Develops a general RKHS framework for metric learning with theoretical guarantees.
result Provides novel generalization guarantees and sample complexity bounds for metric learning in RKHS.
Paper introduces RKHM for more explicit variable structures analysis.
problem Explicitly analyzing structures among variables.
method Orthonormal systems in Hilbert C∗-modules, RKHM. result Theoretical and practical procedures for RKHM orthonormalization.
We study reproducing kernel Hilbert spaces (RKHS) on a Riemannian manifold. In particular, we discuss under which condition Sobolev spaces are RKHS and characterize their reproducing kernels. Further, we introduce and discuss a class of smoother RKHS that we call diffusion spaces. We illustrate the general results with…
This work analyzes the role of data augmentation in self-supervised learning using RKHS approximation and regression.
problem Limited theoretical understanding of the role of data augmentation in self-supervised learning.
method Geometric characterization of the target function given by augmentation, proving generalization bounds.
result Two generalization bounds are derived, one free of model complexity, the other specific to near-optimal encoders.
Ens-CGP synthesizes ensemble-based inference with Gaussian processes.
problem Ensemble-based inference and Gaussian process modeling.
method Formulates Ens-CGP as a conditional Gaussian process for ensemble moments.
result Ens-CGP provides a unified probabilistic foundation for Kalman-type methods.
Unified analysis of kernel-based and locally adaptive bandit optimization methods.
problem Performance of bandit optimization algorithms in RKHS functions.
method Investigates the relationship between kernel regularity and algorithmic performance, characterizing spectral properties of various kernels.
result Unified framework for analyzing kernel-based and locally adaptive bandit algorithms, deriving explicit regret bounds.
Advanced kernels improve Gaussian process accuracy by incorporating domain knowledge.
problem Improving function approximation accuracy in Gaussian processes.
method Advanced kernel designs that enforce specific function properties (symmetry, periodicity) and non-stationarity.
result Advanced kernels significantly enhance function approximation accuracy and relevance.
This paper generalizes regularized regression problems in a hyper-reproducing kernel Hilbert space (hyper-RKHS), illustrates its utility for kernel learning and out-of-sample extensions, and proves asymptotic convergence results for the introduced regression models in an approximation theory view. Algorithmically, we c…
New approach to supervised learning in RKHS and vvRKHS using C∗-algebras.
problem Traditional supervised learning in RKHS and vvRKHS.
method Generalizing supervised learning to RKHM using C∗-algebras. result Constructing RKHMs with enhanced representation power.
Study evaluates RKHS choices for assessing graph models using KSD tests.
problem Effect of RKHS choice on KSD tests for graph model assessment.
method Investigated power performance and computational runtime of KSD tests for ERGMs and synthetic graph generators.
result Different RKHS choices affect KSD test performance and computational runtime.
A new Gaussian process framework uses neural feature maps for scalable, accurate inference.
problem Efficient and accurate Gaussian process inference for diverse data types.
method Neural feature maps to construct expressive kernels, with theoretical guarantees and practical scalability.
result The approach outperforms existing methods in accuracy and efficiency across various data modalities.
Variable selection is central to high-dimensional data analysis, and various algorithms have been developed. Ideally, a variable selection algorithm shall be flexible, scalable, and with theoretical guarantee, yet most existing algorithms cannot attain these properties at the same time. In this article, a three-step va…
Fair representations are a powerful tool for establishing criteria like statistical parity, proxy non-discrimination, and equality of opportunity in learned models. Existing techniques for learning these representations are typically model-agnostic, as they preprocess the original data such that the output satisfies so…
A new method for learning function parameters in operators using data-adaptive RKHS.
problem Learning function parameters in operators with robustness to noise and numerical error.
method Data Adaptive RKHS Tikhonov Regularization (DARTR) method.
result DARTR leads to an accurate estimator robust to noise and numerical error, converging at a consistent rate as data refines.
Study online learning in RKHS with dependent processes, focusing on \(β\)- and \(φ\)-mixing.
problem Online learning in RKHS with dependent data.
method Online regularized learning algorithm in RKHS, analyzing \(β\)- and \(φ\)-mixing sequences.
result Probabilistic upper bounds and convergence rates for mixing coefficients.