Survey of kernels, RKHS, and their applications in machine learning.
problem Understanding kernels and their applications in machine learning.
method Review of historical context, mathematical definitions, and practical applications of kernels.
result Comprehensive overview of kernels, RKHS, and their applications.
Bayesian neural networks with Mercer priors for interpretable uncertainty quantification.
problem Uncertainty quantification in neural networks, especially for complex input-to-output mappings.
method Introducing Mercer priors for BNNs, which approximate a specified GP and are scalable.
result BNNs with Mercer priors can approximate the uncertainty of a specified GP, making them interpretable and scalable.
Study bounds on kernel function entropy for finite measures.
problem Investigate bounds on the ε-entropy of kernel classes.
method Sharp upper and lower bounds for p in [1, +∞] derived from eigenvalue behavior and Mercer series convergence.
result Proves tighter bounds for general kernels compared to previous work.
Develops a framework for learning nonlinear operators using Mercer kernels.
problem Learning nonlinear operators between infinite-dimensional spaces.
method Stochastic approximation framework with Mercer operator-valued kernels.
result Establishes dimension-free polynomial convergence rates for nonlinear operator learning.
Transformers are explained as infinite-dimensional kernel machines.
problem Understanding the mechanics of Transformers in AI.
method Characterized Transformers' attention mechanism as a kernel learning method on Banach spaces.
result Transformer's kernel has infinite feature dimension and can learn any binary non-Mercer reproducing kernel Banach space pair.
Lecture notes on kernel functions and Random Fourier Features.
problem Understanding and approximating kernel functions in machine learning.
method Mathematical background and proofs of concentration results.
result Estimation of error in Random Fourier Features approximation.
The study assesses low-rank approximations in Gaussian Process regression.
problem Improving Gaussian Process regression efficiency with low-rank approximations.
method Analyzes two low-rank approximations: random Fourier features and Mercer expansion truncation.
result Bounds on the divergence and error between exact and approximate GP models.
Kernel interpolation improved with continuous volume sampling.
problem Approximating functions from RKHS using weighted sums of kernel translates.
method Continuous volume sampling for choosing node locations.
result Proved almost optimal bounds for interpolation and quadrature under VS.
The study assesses low-rank approximations in Gaussian Process regression.
problem Improving the efficiency of Gaussian Process regression while maintaining accuracy.
method Analyzes two low-rank approximations: random Fourier features and Mercer expansion truncation, and bounds the divergence and error between exact and approximate models.
result Theoretical bounds on the divergence and error between exact and approximate Gaussian Process models are provided.
Uniform bounds for neural networks' generalization error in overparameterized settings.
problem Generalization error in overparameterized neural networks.
method Neural Tangent kernel theory and Mercer decomposition of the NT kernel in spherical harmonics.
result Uniform generalization bounds for overparameterized neural networks in RKHS.
New method approximates MMD using pseudo-differential operators and singular values.
problem Approximating MMD with pseudo-differential operators and singular values.
method Corresponding pseudo-differential operators to Mercer kernels, approximating p(x,y) with its first r singular values. result The new MMD distance measures the difference of two distributions with respect to r∗ local moments, where r∗ depends on singular values decay rate. Paper identifies key function spaces for ReLU networks based on Fisher information.
problem Understanding the structure of Fisher information matrices in ReLU networks.
method Spectral decomposition of Fisher information matrices, focusing on the first three eigenspaces.
result The first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, corresponding to spherical harmonic functions of order ≤2.
Overlapping clustering problem is an important learning issue in which clusters are not mutually exclusive and each object may belongs simultaneously to several clusters. This paper presents a kernel based method that produces overlapping clusters on a high feature space using mercer kernel techniques to improve separa…
Devoted to multi-task learning and structured output learning, operator-valued kernels provide a flexible tool to build vector-valued functions in the context of Reproducing Kernel Hilbert Spaces. To scale up these methods, we extend the celebrated Random Fourier Feature methodology to get an approximation of operator-…
Interest in multioutput kernel methods is increasing, whether under the guise of multitask learning, multisensor networks or structured output data. From the Gaussian process perspective a multioutput Mercer kernel is a covariance function over correlated output functions. One way of constructing such kernels is based …
As a robust nonlinear similarity measure in kernel space, correntropy has received increasing attention in domains of machine learning and signal processing. In particular, the maximum correntropy criterion (MCC) has recently been successfully applied in robust regression and filtering. The default kernel function in c…
New learning rates derived for Tikhonov-regularized problems without kernel assumptions.
problem Learning rates for Tikhonov-regularized learning problems.
method Minimax adaptive rates derived using Fourier isocapacitary condition and interpolation theory.
result Derivation of minimax adaptive rates without requiring kernel assumptions.
Recently, there has been emerging interest in constructing reproducing kernel Banach spaces (RKBS) for applied and theoretical purposes such as machine learning, sampling reconstruction, sparse approximation and functional analysis. Existing constructions include the reflexive RKBS via a bilinear form, the semi-inner-p…
In machine learning or statistics, it is often desirable to reduce the dimensionality of a sample of data points in a high dimensional space Rd. This paper introduces a dimensionality reduction method where the embedding coordinates are the eigenvectors of a positive semi-definite kernel obtained as the sol…
New asymmetric kernel methods improve feature learning.
problem Improving feature learning with asymmetric kernels.
method Coupled covariance eigenproblem and Nyström method.
result Empirical evaluations show benefits of KSVD.
This paper presents a unified framework to tackle estimation problems in Digital Signal Processing (DSP) using Support Vector Machines (SVMs). The use of SVMs in estimation problems has been traditionally limited to its mere use as a black-box model. Noting such limitations in the literature, we take advantage of sever…
Paper learns optimal kernels for Gaussian process regression in aerodynamics.
problem Approximating complex functions from limited data in aerodynamics.
method Two algorithms: Kernel Flow and Spectral Kernel Ridge Regression.
result Explicit construction of optimal kernels based on target function features.
A new deep neural network tackles nonlinear functional regression with improved dimensionality reduction.
problem Nonlinear functional regression in infinite-dimensional functional data analysis.
method Functional deep neural network with adaptive kernel embedding and projection steps.
result Explicit rates of approximating nonlinear smooth functionals are derived, and the network is shown to be effective in both simulated and real datasets.
In data science, determining proximity between observations is critical to many downstream analyses such as clustering, information retrieval and classification. However, when the underlying structure of the data probability space is unclear, the function used to compute similarity between data points is often arbitrar…
This work analyzes how different layers in deep neural networks contribute to generalization error.
problem Understanding the role of each layer in deep neural networks for generalization.
method Spectral analysis, Neural Tangent Kernel, Hermite polynomials, Spherical Harmonics.
result Initial layers in deep neural networks have a larger bias towards high-frequency functions.
Unified framework for constructing kernels for transport equations and Koopman eigenfunctions.
problem Constructing kernels for transport equations and Koopman eigenfunctions.
method Three methods: variational principle, Green's function, and resolvent operator.
result Kernels constructed via these methods are identical under mild assumptions.
Study on Neural Tangent Kernel of Matrix Product States and their convergence.
problem Understanding the convergence of Neural Tangent Kernel of Matrix Product States.
method Analyzing the Neural Tangent Kernel of Matrix Product States and proving its convergence in the infinite bond dimensional limit.
result The Neural Tangent Kernel of Matrix Product States converges to a constant matrix during training.
We investigate a generic problem of learning pairwise exponential family graphical models with pairwise sufficient statistics defined by a global mapping function, e.g., Mercer kernels. This subclass of pairwise graphical models allow us to flexibly capture complex interactions among variables beyond pairwise product. …
Positive weights improve kernel quadrature's accuracy.
problem Improving kernel quadrature weights to be positive and stable.
method Using convex geometry to approximate the kernel mean embedding with positive weights.
result Positive weights lead to improved kernel quadrature bounds with Monte-Carlo-beating rates.
Paper develops a new algorithm for distribution regression with optimal learning rates.
problem Distribution regression with limited second-stage samples.
method Multi-penalty regularization in a reproducing kernel Hilbert space.
result Derives optimal learning rates for distribution regression.
A new approach to cost-sensitive multiclass classification prioritizes certain classes over others.
problem Cost-sensitive multiclass classification where some classes are more important than others.
method Apportioned margin framework that shifts the decision boundary to prioritize certain classes.
result The method improves the error rate for important classes while reducing overall error.
Sequential modelling with self-attention has achieved cutting edge performances in natural language processing. With advantages in model flexibility, computation complexity and interpretability, self-attention is gradually becoming a key component in event sequence models. However, like most other sequence models, self…
A new PCA method for analyzing point processes.
problem Analyzing variability in replicated point processes.
method Functional Principal Component Analysis (fPCA) on cumulative mass functions.
result Established convergence and introduced principal measures.
We reformulate unsupervised dimension reduction problem (UDR) in the language of tempered distributions, i.e. as a problem of approximating an empirical probability density function by another tempered distribution, supported in a k-dimensional subspace. We show that this task is connected with another classical prob…
A simple framework Probabilistic Multi-view Graph Embedding (PMvGE) is proposed for multi-view feature learning with many-to-many associations so that it generalizes various existing multi-view methods. PMvGE is a probabilistic model for predicting new associations via graph embedding of the nodes of data vectors with …
Many unsupervised kernel methods rely on the estimation of the kernel covariance operator (kernel CO) or kernel cross-covariance operator (kernel CCO). Both kernel CO and kernel CCO are sensitive to contaminated data, even when bounded positive definite kernels are used. To the best of our knowledge, there are few well…
To the best of our knowledge, there are no general well-founded robust methods for statistical unsupervised learning. Most of the unsupervised methods explicitly or implicitly depend on the kernel covariance operator (kernel CO) or kernel cross-covariance operator (kernel CCO). They are sensitive to contaminated data, …
Deep neural kernels and Laplace kernel have equivalent RKHS on spheres.
problem Comparing RKHS of deep neural tangent and Laplace kernels.
method Proof of RKHS equivalence using sphere restrictions and kernel properties.
result RKHS of deep neural tangent kernel and Laplace kernel are the same on Sd−1. Kernel methods linked to feature subspaces and maximal correlation kernels.
problem Understanding kernel methods and their relationship to feature extraction.
method Established a correspondence between feature subspaces and kernels, introduced maximal correlation kernels, and demonstrated their optimality.
result Kernel SVM on maximal correlation kernel achieves minimum prediction error.
PGF kernels analyze spherical data using generalized RBF kernels.
problem Analysis of spherical data.
method Introduced PGF kernels and a semi-parametric learning algorithm.
result PGF kernels generalize RBF kernels for spherical data.
Adapts manifold structure for better clustering performance.
problem Lack of consideration for local manifold structure in existing multiple kernel k-means methods.
method Adopts manifold adaptive kernel to integrate local manifold structure of kernels.
result Proposed method outperforms state-of-the-art methods.
Optimal kernel in KR can be data-dependent, improving model performance.
problem Fixed kernel in KR limits model performance.
method Considered data-dependent kernels for KR, using posterior covariance.
result Data-dependent kernel choice leads to optimal performance.
Quantum kernels can be efficiently embedded into classical feature spaces.
problem Can all quantum kernels be efficiently embedded into classical feature spaces?
method Invoking computational universality and using techniques like random Fourier features, the authors show that certain classes of quantum kernels can be efficiently embedded.
result For shift-invariant and composition kernels, embedding quantum kernels are universal and efficient.
New random feature maps for Laplacian and related kernels.
problem Challenges in approximating the Laplacian kernel and its generalizations.
method Developed random feature maps for Laplacian and related kernels, providing efficient sampling schemes.
result Demonstrated the efficacy of these random feature maps on real datasets.
New method for learning with non-Euclidean data using decomposable kernels.
problem Difficulty in using classical kernels for non-Euclidean data.
method Reproducing kernel Krein space (RKKS) methods for kernels that admit a positive decomposition.
result Invariant kernels can be used for learning in non-Euclidean spaces.
New estimator reduces kernel mean estimation error.
problem Kernel mean estimation in reproducing kernel Hilbert spaces.
method Corrupt data with known distributions and estimate kernel mean under the corrupted distribution.
result The marginalized kernel mean estimator achieves lower estimation error.
Estimates kernel eigenvalues for compositional dot-product kernels.
problem Improving estimates for kernel eigenvalues.
method Eigenvalue decay estimates of integral operators associated with dot-product kernels.
result Improved estimates for kernel volumes in reproducing kernel Hilbert spaces.
Optimal Biweight kernel and computationally efficient Epanechnikov kernel for modal linear regression.
problem Finding the best kernel for modal linear regression.
method Refined analysis of asymptotic statistical behavior and IRLS algorithm convergence.
result Biweight kernel minimizes asymptotic mean squared error, Epanechnikov kernel guarantees IRLS convergence.