Two EM algorithms estimate prior distributions in mixture of linear regressions.
problem Estimating prior distributions in mixture of linear regressions.
method Two EM algorithms: one for continuous priors, one for discrete priors.
result Both algorithms accurately estimate prior distributions and the number of clusters.
NPMLE estimator automatically chooses the right model complexity for Gaussian mixtures.
problem Learning mixture models and empirical Bayes estimation with non-convex likelihoods.
method Nonparametric maximum likelihood estimator (NPMLE) using complex-analytic techniques.
result NPMLE solution has O(logn) atoms with high probability, improving model complexity. We study the Nonparametric Maximum Likelihood Estimator (NPMLE) for estimating Gaussian location mixture densities in d-dimensions from independent observations. Unlike usual likelihood-based methods for fitting mixtures, NPMLEs are based on convex optimization. We prove finite sample results on the Hellinger accurac…
The paper provides stability guarantees for non-parametric maximum likelihood estimation using statistical mechanics.
problem Non-parametric maximum likelihood estimation and Gaussian mixture models.
method Statistical mechanics analysis to establish stability guarantees for NPMLE.
result High probability upper bounds on the Kullback-Leibler divergence between NPMLE estimators and the true density.
New method uses SURE to denoise signals, outperforming NPMLE.
problem Learning to optimally denoise signals corrupted by Gaussian noise.
method Hyvärinen's score matching (SM) is shown equivalent to SURE minimization.
result SURE achieves nearly parametric rates of convergence in empirical Bayes settings.
Paper derives convergence rates for NPMLE in Hellinger distance using deep neural networks.
problem Difficulty in proving convergence of excess risk in nonparametric logistic regression.
method Unified approach for analyzing NPMLE, deriving convergence rates in Hellinger distance.
result Derives nearly optimal convergence rates for NPMLE with deep neural networks.
Transformers solve Poisson means estimation via empirical Bayes.
problem Estimating Poisson means under empirical Bayes setting.
method Pre-trained transformer learns to adapt to unknown prior and do in-context learning.
result Transformers achieve vanishing regret with large models and outperform classical algorithms.
Paper improves matrix-valued data classification using nonparametric LDA.
problem Classification of matrix-valued data in neuroimaging and signal processing.
method Nonparametric LDA based on NPMLE for vectorized and scaled matrices.
result Improves classification performance across various data structures.
The paper improves Fisher-Pitman tests for Poisson mixtures, detecting autism-related genes.
problem Detecting differentially expressed genes between autism and control subjects.
method Nonparametric Poisson mixtures and Fisher-Pitman permutation tests.
result The tests reveal genes missed by common methods, demonstrating rate optimality.
New method for high-dimensional linear regression using empirical Bayes.
problem Estimating prior in high-dimensional linear regression.
method Variational empirical Bayes approach with NPMLE and mean field approximation.
result Established asymptotic consistency and computational efficiency of the method.
New algorithm for fitting Gaussian mixtures using Wasserstein-Fisher-Rao geometry.
problem Hard problem of fitting Gaussian mixture models to data computationally.
method Gradient descent over Wasserstein-Fisher-Rao geometry for probability measures.
result Established convergence guarantees for the proposed algorithm.
Empirical Bayes method improves Gaussian sequence model inference.
problem Estimating parameters in correlated Gaussian sequence models.
method Maximum Composite Marginal Likelihood (CML) estimator, leveraging geometric Brascamp-Lieb inequality.
result CML estimator converges at rate \( n_*^{-1/2} \) in weighted Hellinger distance.
Many unsupervised kernel methods rely on the estimation of the kernel covariance operator (kernel CO) or kernel cross-covariance operator (kernel CCO). Both kernel CO and kernel CCO are sensitive to contaminated data, even when bounded positive definite kernels are used. To the best of our knowledge, there are few well…
Survey of kernels, RKHS, and their applications in machine learning.
problem Understanding kernels and their applications in machine learning.
method Review of historical context, mathematical definitions, and practical applications of kernels.
result Comprehensive overview of kernels, RKHS, and their applications.
To the best of our knowledge, there are no general well-founded robust methods for statistical unsupervised learning. Most of the unsupervised methods explicitly or implicitly depend on the kernel covariance operator (kernel CO) or kernel cross-covariance operator (kernel CCO). They are sensitive to contaminated data, …
Deep neural kernels and Laplace kernel have equivalent RKHS on spheres.
problem Comparing RKHS of deep neural tangent and Laplace kernels.
method Proof of RKHS equivalence using sphere restrictions and kernel properties.
result RKHS of deep neural tangent kernel and Laplace kernel are the same on Sd−1. Kernel methods linked to feature subspaces and maximal correlation kernels.
problem Understanding kernel methods and their relationship to feature extraction.
method Established a correspondence between feature subspaces and kernels, introduced maximal correlation kernels, and demonstrated their optimality.
result Kernel SVM on maximal correlation kernel achieves minimum prediction error.
PGF kernels analyze spherical data using generalized RBF kernels.
problem Analysis of spherical data.
method Introduced PGF kernels and a semi-parametric learning algorithm.
result PGF kernels generalize RBF kernels for spherical data.
Adapts manifold structure for better clustering performance.
problem Lack of consideration for local manifold structure in existing multiple kernel k-means methods.
method Adopts manifold adaptive kernel to integrate local manifold structure of kernels.
result Proposed method outperforms state-of-the-art methods.
Optimal kernel in KR can be data-dependent, improving model performance.
problem Fixed kernel in KR limits model performance.
method Considered data-dependent kernels for KR, using posterior covariance.
result Data-dependent kernel choice leads to optimal performance.
Quantum kernels can be efficiently embedded into classical feature spaces.
problem Can all quantum kernels be efficiently embedded into classical feature spaces?
method Invoking computational universality and using techniques like random Fourier features, the authors show that certain classes of quantum kernels can be efficiently embedded.
result For shift-invariant and composition kernels, embedding quantum kernels are universal and efficient.
New random feature maps for Laplacian and related kernels.
problem Challenges in approximating the Laplacian kernel and its generalizations.
method Developed random feature maps for Laplacian and related kernels, providing efficient sampling schemes.
result Demonstrated the efficacy of these random feature maps on real datasets.
New method for learning with non-Euclidean data using decomposable kernels.
problem Difficulty in using classical kernels for non-Euclidean data.
method Reproducing kernel Krein space (RKKS) methods for kernels that admit a positive decomposition.
result Invariant kernels can be used for learning in non-Euclidean spaces.
New estimator reduces kernel mean estimation error.
problem Kernel mean estimation in reproducing kernel Hilbert spaces.
method Corrupt data with known distributions and estimate kernel mean under the corrupted distribution.
result The marginalized kernel mean estimator achieves lower estimation error.
We present Random Partition Kernels, a new class of kernels derived by demonstrating a natural connection between random partitions of objects and kernels between those objects. We show how the construction can be used to create kernels from methods that would not normally be viewed as random partitions, such as Random…
The NNGP kernel's predictions closely match those of the Matern kernel under certain conditions.
problem Comparing NNGP kernels to Matern kernels in practical applications.
method Demonstrated the necessity of normalization for NNGP kernels, explored numerical challenges, and compared predictions and performance.
result NNGP kernel predictions closely match Matern kernel predictions under specific circumstances.
In this paper, we compare 5 different nonlinear kernels: min-max, RBF, fRBF (folded RBF), acos, and acos-χ2, on a wide range of publicly available datasets. The proposed fRBF kernel performs very similarly to the RBF kernel. Both RBF and fRBF kernels require an important tuning parameter (γ). Interestingly, for a …
New kernels allow learning from non-separable data.
problem Learning from non-separable data.
method Introducing entangled kernels and a two-step algorithm.
result Efficient algorithm for learning entangled kernels.
Laplace kernel and Neural Tangent Kernels are shown to be nearly identical for normalized data.
problem Understanding the similarity between Laplace and Neural Tangent Kernels.
method Theoretical analysis and experiments on normalized data.
result Laplace kernel and Neural Tangent Kernels have nearly identical eigenfunctions and RKHS for normalized data.
Kernel smoothing on unknown manifolds with bounds and asymptotic normality.
problem Data on unknown manifolds without boundaries.
method Finite sample bounds and asymptotic normality for kernel smoothing and its derivatives.
result Established finite sample bounds and asymptotic normality for kernel smoothing.
In this paper we propose a family of tractable kernels that is dense in the family of bounded positive semi-definite functions (i.e. can approximate any bounded kernel with arbitrary precision). We start by discussing the case of stationary kernels, and propose a family of spectral kernels that extends existing approac…
The success of kernel-based learning methods depend on the choice of kernel. Recently, kernel learning methods have been proposed that use data to select the most appropriate kernel, usually by combining a set of base kernels. We introduce a new algorithm for kernel learning that combines a {\em continuous set of base …
New kernels capture both local and non-local interactions efficiently.
problem Designing kernels that capture both local and non-local interactions while remaining computationally tractable.
method Spectral truncation kernels based on C∗-algebra. result Spectral truncation kernels induce interactions across the data function domain and reduce computational cost.
Quantum kernel machines need to use more complex kernels to fully exploit their potential.
problem Current quantum kernels struggle with complex learning tasks due to limited degrees of freedom.
method Propose using operator-valued kernels and C∗-algebraic representations to enhance quantum kernels. result Quantum operator-valued kernels can reveal structural dependencies that scalar-valued kernels miss.
Constructing the adjacency graph is fundamental to graph-based clustering. Graph learning in kernel space has shown impressive performance on a number of benchmark data sets. However, its performance is largely determined by the chosen kernel matrix. To address this issue, the previous multiple kernel learning algorith…
Optimal kernel improves estimation accuracy in modal statistical methods.
problem Estimation accuracy of kernel-based modal statistical methods depends on the kernel used.
method The study theoretically shows an optimal kernel that minimizes asymptotic error criterion.
result An optimal kernel minimizes the error criterion when using an optimal bandwidth.
Efficiently searches through Gaussian process kernels using symbolic representation and Bayesian optimization.
problem Manual selection of kernels in Gaussian processes is complex and computationally expensive.
method Proposes a novel method using symbolic representation and Bayesian optimization to search through a structured kernel space.
result Empirically shows a computationally more efficient way of searching through a discrete kernel space.
The term "CoRE kernel" stands for correlation-resemblance kernel. In many applications (e.g., vision), the data are often high-dimensional, sparse, and non-binary. We propose two types of (nonlinear) CoRE kernels for non-binary sparse data and demonstrate the effectiveness of the new kernels through a classification ex…
MKLpy simplifies Multiple Kernel Learning in Python.
problem Learning optimal kernel functions from data.
method Python-based framework for Multiple Kernel Learning algorithms.
result Maximizes usability and simplifies development of novel solutions.
Sparse Kernel Flows learns dynamical systems from data.
problem Learning dynamical systems from limited data.
method Sparse Kernel Flows: trains optimal kernel from a dictionary of kernels.
result Sparse Kernel Flows can learn from 132 chaotic systems.
The study investigates kernel-target alignment in tree ensemble kernels.
problem The degree of kernel-target alignment affects the performance of tree ensemble kernels in kernel learning.
method Eigenanalysis of the kernel matrix and sensitivity analysis via landmark learning.
result Good performance of tree ensemble kernels is associated with strong kernel-target alignment.
Many real world graphs, such as the graphs of molecules, exhibit structure at multiple different scales, but most existing kernels between graphs are either purely local or purely global in character. In contrast, by building a hierarchy of nested subgraphs, the Multiscale Laplacian Graph kernels (MLG kernels) that we …
Two adaptive kernel selection methods improve the accuracy of Kernelized Diffusion Maps.
problem Selecting an appropriate kernel for Kernelized Diffusion Maps.
method Two complementary approaches: variational outer loop and unsupervised cross-validation.
result Both methods improve the quality and stability of the recovered eigenfunctions.
This study examines the practical equivalence of Laplace and neural tangent kernels.
problem Understanding the practical equivalence of Laplace and neural tangent kernels.
method The study matches the kernels exactly and by matching posteriors of a Gaussian process. It also analyzes the kernels in R^d and experiments with them in regression tasks.
result The Laplace and neural tangent kernels are practically equivalent.
The paper provides consistency results for KDE on manifolds with irregular kernels.
problem Analyzing density estimation on manifolds with complex kernels.
method Strong uniform consistency with rates for KDE on Riemannian manifolds with Riemann integrable kernels.
result Strong uniform consistency with rates for KDE on manifolds.
Kernel methods are studied in a mean field limit for high-dimensional data.
problem Analyzing kernel methods in high-dimensional data with many variables.
method Investigation of kernel methods in the mean field limit of interacting particle systems.
result Rigorous mean field limit of kernels and detailed analysis of the limiting reproducing kernel Hilbert space.
New quantum kernels avoid overfitting by combining local and global components.
problem Exponential concentration in quantum kernels leads to overfitting.
method Local-global quantum kernels combining small subsystem and full-system measurements.
result Demonstrated benign overfitting in local-global quantum kernels.
IDK improves anomaly detection for points and groups without explicit learning.
problem Anomaly detection for points and groups using kernel methods.
method Isolation Distributional Kernel (IDK) addresses data independence and intractable dimensionality issues.
result IDK outperforms existing methods for both point and group anomaly detection.