The paper establishes concentration bounds for embeddings of generative models.
problem Analyzing statistical properties of generative models.
method High probability concentration bounds on sample vector embeddings using Data Kernel Perspective Space.
result Determines the number of samples needed for accurate approximation of generative model embeddings.
Kernel-based Bayesian filter for nonlinear systems using infinite-dimensional operators.
problem Modeling and predicting nonlinear dynamical systems.
method Functional Bayesian perspective, reproducing kernel Hilbert space, Gaussian kernel.
result Effective approximation and accurate results for nonlinear systems.
Although operator-valued kernels have recently received increasing interest in various machine learning and functional data analysis problems such as multi-task learning or functional regression, little attention has been paid to the understanding of their associated feature spaces. In this paper, we explore the potent…
This work incorporates the multi-modality of the data distribution into a Gaussian Process regression model. We approach the problem from a discriminative perspective by learning, jointly over the training data, the target space variance in the neighborhood of a certain sample through metric learning. We start by using…
Kernel methods are among the most popular techniques in machine learning. From a frequentist/discriminative perspective they play a central role in regularization theory as they provide a natural choice for the hypotheses space and the regularization functional through the notion of reproducing kernel Hilbert spaces. F…
New approach to supervised learning in RKHS and vvRKHS using C∗-algebras.
problem Traditional supervised learning in RKHS and vvRKHS.
method Generalizing supervised learning to RKHM using C∗-algebras. result Constructing RKHMs with enhanced representation power.
Develops robust persistence diagrams using kernel methods.
problem Persistence diagrams are sensitive to data perturbations.
method Constructs robust persistence diagrams from superlevel filtrations of robust density estimators using reproducing kernels.
result Robust persistence diagrams are consistent estimators in bottleneck distance.
Develops RKHS framework for analyzing tree ensembles.
problem Analyzing the theoretical properties of tree ensembles.
method Reproducing Kernel Hilbert Spaces (RKHS) for tree ensembles.
result Characterizes Random Forest predictor as minimizer of a penalized empirical risk functional in RKHS.
Optimal recovery framework for non-IID data in Hilbert spaces.
problem Generalization in non-IID data scenarios.
method Optimal recovery perspective, semidefinite programming, kernel ridgeless regression.
result Optimal recovery formula coincides with kernel ridgeless regression in some cases.
We present a new framework for online Least Squares algorithms for nonlinear modeling in RKH spaces (RKHS). Instead of implicitly mapping the data to a RKHS (e.g., kernel trick), we map the data to a finite dimensional Euclidean space, using random features of the kernel's Fourier transform. The advantage is that, the …
Gradient descent in neural networks analyzed using RKBS for broader applicability.
problem Analyzing neural network training in the over-parametrized limit.
method Constructing an exact power-series representation of neural networks in RKBS, proving replicability of gradient descent sequences.
result Gradient descent sequences can be exactly replicated by regularized sequential learning in RKBS, providing new theoretical insights.
A new method for adapting to label shifts using class probability matching.
problem Adapting to label shifts where class probabilities differ between source and target domains.
method Class Probability Matching using Kernel Methods (CPMKM) framework.
result CPMKM outperforms existing methods on real datasets.
Researchers approximate conditional expectation operators using kernel methods.
problem Statistical approximation of conditional expectation operators under minimal assumptions.
method Modifying the domain of the operator, approximating it by Hilbert-Schmidt operators in a reproducing kernel Hilbert space.
result The nonparametric estimate of the operator converges to a specific limiting object.
Abstract perspective on quadratic programming for optimal portfolio allocation.
problem Optimal allocation problems in long portfolio theory.
method Using maximum principles and distinguished boundaries in reproducing kernel Hilbert spaces.
result Support of an optimal distribution lies in a variety intersecting a distinguished boundary.
Unified theory for adaptive image convolutions using metric perspectives.
problem Fixed kernels in convolutions limit adaptability in image processing.
method Metric perspective on images as 2D manifolds with local distances, proposing metric convolutions.
result Metric convolutions provide better generalisation and competitive performance.
DKPS provides guarantees for synthetic data from Transformer models, improving downstream tasks.
problem Lack of labeled data for building performant AI models.
method Data Kernel Perspective Space (DKPS) for mathematical analysis of synthetic data quality.
result Concrete statistical guarantees for the quality of transformer model outputs.
Two-layer neural networks learn efficiently using kernel methods in mean-field analysis.
problem Feature learning ability of two-layer neural networks in the mean-field regime.
method Mean-field analysis through kernel methods, focusing on dynamics of the first layer's kernel.
result Two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than kernel methods.
Gradient descent reshapes the function space of neural networks.
problem Understanding how feature learning affects the function space of neural networks.
method Characterized the evolution of the feature space during training using a two-layer neural network.
result Gradient descent induces a data-adaptive deformation that selectively enhances signal-aligned directions.
Transformers are explained as infinite-dimensional kernel machines.
problem Understanding the mechanics of Transformers in AI.
method Characterized Transformers' attention mechanism as a kernel learning method on Banach spaces.
result Transformer's kernel has infinite feature dimension and can learn any binary non-Mercer reproducing kernel Banach space pair.
Bayesian kernel regression improves functional output prediction.
problem Functional output regression in supervised learning.
method Kernel methods, leveraging covariance structure within function values.
result Enhanced prediction accuracy and handling of high-dimensional nonlinearity.
The paper provides concentration bounds for embeddings of generative models.
problem Establishing accurate statistical analysis of generative models.
method Data Kernel Perspective Space embedding method.
result Required number of sample responses for accurate approximation.
Representation costs in data science: Unifying function-space views of parametric methods
problem Analyzing representation costs of parametric data-fitting methods
method Developing a general framework for analyzing representation costs through parameter-space regularizers
result Proving that many natural results hold in this abstract setting, including representer theorems for parametric methods on their native spaces
We propose a new point of view for regularizing deep neural networks by using the norm of a reproducing kernel Hilbert space (RKHS). Even though this norm cannot be computed, it admits upper and lower approximations leading to various practical strategies. Specifically, this perspective (i) provides a common umbrella f…
Quantum models are rephrased as kernel methods, improving performance.
problem Improving quantum machine learning models by encoding data into quantum states.
method Rephrasing quantum models as kernel methods and using support vector machines.
result Kernel-based training finds better quantum models than variational circuit training.
Sketching accelerates structured prediction methods for large datasets.
problem Scaling surrogate kernel methods for large datasets.
method Sketching approximations applied to input and output feature maps.
result Achieves close-to-optimal rates with reduced sketch size.
Mode decomposition is a prototypical pattern recognition problem that can be addressed from the (a priori distinct) perspectives of numerical approximation, statistical inference and deep learning. Could its analysis through these combined perspectives be used as a Rosetta stone for deciphering mechanisms at play in de…
We study the problem of structured output learning from a regression perspective. We first provide a general formulation of the kernel dependency estimation (KDE) problem using operator-valued kernels. We show that some of the existing formulations of this problem are special cases of our framework. We then propose a c…
Cross-validation methods help learn dynamical systems from data.
problem Learning surrogate models for dynamical systems from limited data.
method Variants of cross-validation (Kernel Flows, MMD, Lyapunov exponents).
result Simple approaches for kernel selection in dynamical system emulators.
We present a simple but powerful reinterpretation of kernelized locality-sensitive hashing (KLSH), a general and popular method developed in the vision community for performing approximate nearest-neighbor searches in an arbitrary reproducing kernel Hilbert space (RKHS). Our new perspective is based on viewing the step…
A new kernel improves Gaussian process performance for non-stationary data.
problem Poor prediction and uncertainty quantification with standard GPs.
method Study and comparison of non-stationary kernels, propose a new combined kernel.
result A new kernel outperforms existing stationary and non-stationary kernels.
Kernel methods benefit from enforcing invariance, reducing generalization error.
problem Improving generalization in kernel methods.
method Function space perspective and feature averaging for invariance.
result Strict non-zero generalization benefit for kernel ridge regression with invariant targets.
This study connects Gaussian processes and RKHS, bridging two machine learning communities.
problem Understanding the relationship between Gaussian processes and RKHS.
method Examining connections and equivalences in regression, interpolation, and other topics.
result Established the equivalence between Gaussian Hilbert space and RKHS.
Gaussian Processes (GPs) provide a general and analytically tractable way of modeling complex time-varying, nonparametric functions. The Automatic Bayesian Covariance Discovery (ABCD) system constructs natural-language description of time-series data by treating unknown time-series data nonparametrically using GP with …
Deep convolutional networks can be understood through kernel methods, providing insights into their inductive bias.
problem Understanding the functional space and inductive bias of deep convolutional networks.
method Using kernel methods to analyze simple hierarchical kernels with convolution and pooling layers.
result The RKHS consists of additive models of interaction terms between patches, and pooling layers encourage spatial similarities.
SGD converges to optimal solution in perfect data fitting problem.
problem Noiseless least-squares with perfect data fitting.
method Last iterate of SGD with constant step-size.
result Convergence of SGD final iterate for non-strongly convex problem.
Enhances graph neural networks by considering feature similarities in node aggregation.
problem Ignoring node feature similarities in traditional graph aggregation schemes.
method Interprets node aggregation as kernel weighting, proposing a framework that considers feature similarities.
result Proposed framework outperforms traditional GCNs in real-world applications.
In this paper we first identify a basic limitation in gradient descent-based optimization methods when used in conjunctions with smooth kernels. An analysis based on the spectral properties of the kernel demonstrates that only a vanishingly small portion of the function space is reachable after a polynomial number of g…
Modeling videos and image-sets as linear subspaces has proven beneficial for many visual recognition tasks. However, it also incurs challenges arising from the fact that linear subspaces do not obey Euclidean geometry, but lie on a special type of Riemannian manifolds known as Grassmannian. To leverage the techniques d…
This paper improves neural network generalization by dynamically learning kernel parameters.
problem Improving neural network generalization and adaptability.
method Diagonal adaptive kernel model that learns kernel eigenvalues and output coefficients during training.
result The diagonal adaptive kernel model significantly improves generalization over fixed-kernel methods.
Overparametrized neural networks can generalize well with proper regularization.
problem Generalization guarantee for noisy data in overparametrized neural networks.
method Nonparametric analysis of ℓ2-regularized GD trajectories. result Achieving minimax optimal rate of L2 estimation error with ℓ2 regularization. Study evaluates posterior covariance matrix W for frequentist evaluation of Bayesian estimators.
problem Evaluating variability of posterior estimates in Bayesian models.
method Use of Bayesian Infinitesimal Jackknife approximation and W-kernel.
result Principal space of W is central to frequentist evaluation of Bayesian models.
This paper shows equivalence between SVGD and BBVI using kernel gradient flows.
problem Bayesian inference methods and their equivalence.
method Formalizes equivalence between SVGD and BBVI using kernel gradient flows.
result BBVI corresponds precisely to SVGD when using the neural tangent kernel.
We focus on kernel methods for set-valued inputs and their application to Bayesian set optimization, notably combinatorial optimization. We investigate two classes of set kernels that both rely on Reproducing Kernel Hilbert Space embeddings, namely the ``Double Sum'' (DS) kernels recently considered in Bayesian set opt…
We analyze in this paper a random feature map based on a theory of invariance I-theory introduced recently. More specifically, a group invariant signal signature is obtained through cumulative distributions of group transformed random projections. Our analysis bridges invariant feature learning with kernel methods, as …
Sparse Kernel Flows learns dynamical systems from data.
problem Learning dynamical systems from limited data.
method Sparse Kernel Flows: trains optimal kernel from a dictionary of kernels.
result Sparse Kernel Flows can learn from 132 chaotic systems.
We investigate time-dependent data analysis from the perspective of recurrent kernel machines, from which models with hidden units and gated memory cells arise naturally. By considering dynamic gating of the memory cell, a model closely related to the long short-term memory (LSTM) recurrent neural network is derived. E…
Spectral algorithms on manifolds using diffusion kernels improve convergence rates.
problem The limitations of existing spectral algorithms in RKHSs for data on manifolds.
method Integrating manifold structure into spectral algorithms using heat kernel diffusion spaces.
result Spectral algorithms converge to the target function and its derivatives in a strong sense, with rates dependent on manifold intrinsic dimension.
Unified view of GP approximations improves efficiency.
problem Disparate variational features limit GP efficiency.
method View GP as a Banach space to unify feature selection.
result Unified understanding of existing and new features.