The neural tangent kernel equivalence theorem fails in practice.
problem Does the neural tangent kernel (NTK) equivalence theorem hold in practical neural network training?
method Rigorously derived NTK and conducted numerical experiments to evaluate the equivalence theorem.
result Adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in predictor error.
This study examines the practical equivalence of Laplace and neural tangent kernels.
problem Understanding the practical equivalence of Laplace and neural tangent kernels.
method The study matches the kernels exactly and by matching posteriors of a Gaussian process. It also analyzes the kernels in R^d and experiments with them in regression tasks.
result The Laplace and neural tangent kernels are practically equivalent.
Study shows deterministic equivalent for neural network kernel convergence.
problem Understanding convergence of neural network kernels.
method Analyzes empirical spectral distribution of Conjugate Kernel, proving convergence to a deterministic limit.
result Obtains a deterministic equivalent for the Stieltjes transform and resolvent of the Conjugate Kernel.
Kernel tests assess equivalence between distributions without assuming specific moments.
problem Traditional goodness-of-fit tests fail to detect meaningful distributional differences.
method Proposes kernel-based tests using kernel Stein discrepancy and Maximum Mean Discrepancy.
result Tests assess the absence of meaningful distributional differences under controlled error rates.
This study connects Gaussian processes and RKHS, bridging two machine learning communities.
problem Understanding the relationship between Gaussian processes and RKHS.
method Examining connections and equivalences in regression, interpolation, and other topics.
result Established the equivalence between Gaussian Hilbert space and RKHS.
New insights into neural network kernels for rectified MLPs.
problem Understanding the behavior of weight distributions in rectified MLPs.
method Deriving equivalent kernels for MLPs with ReLU or Leaky ReLU activations under various weight distributions.
result Kernels corresponding to layers with certain weight distributions are asymptotically universal and well approximated by Gaussian kernels.
The purpose of this paper is to introduce a concept of equivalence between machine learning algorithms. We define two notions of algorithmic equivalence, namely, weak and strong equivalence. These notions are of paramount importance for identifying when learning prop erties from one learning algorithm can be transferre…
This paper shows equivalence between SVGD and BBVI using kernel gradient flows.
problem Bayesian inference methods and their equivalence.
method Formalizes equivalence between SVGD and BBVI using kernel gradient flows.
result BBVI corresponds precisely to SVGD when using the neural tangent kernel.
Paper studies the theoretical equivalence between implicit and explicit neural networks in high dimensions.
problem Lack of theoretical analysis of implicit and explicit neural networks.
method Examined high-dimensional implicit neural networks and established their equivalence to explicit networks.
result Equivalence between implicit and explicit neural networks in high dimensions.
Study reveals an equivalence principle for the spectrum of random inner-product kernel matrices in polynomial scaling.
problem Understanding the spectrum of random kernel matrices in polynomial scaling regimes.
method Investigates random matrices with nonlinear kernel functions applied to inner products of uniformly distributed vectors.
result The spectrum of the random kernel matrix is asymptotically equivalent to a simpler matrix model through free additive convolution.
Study of eigenvalues in nonlinear kernels for classification of separable data.
problem Understanding the applicability of linear equivalents in nonlinearly separable data classification.
method Analysis of conjugate kernels and their quadratic equivalents for a canonical nonlinearly separable dataset (XOR problem).
result Identification of regimes where nonlinear kernels deviate from linear equivalents, leading to label-aligned eigenspaces.
Associating genetic markers with a multidimensional phenotype is an important yet challenging problem. In this work, we establish the equivalence between two popular methods: kernel-machine regression (KMR), and kernel distance covariance (KDC). KMR is a semiparametric regression frameworks that models the covariate ef…
This work establishes the equivalence between neural networks and support vector machines.
problem Establishing the equivalence between neural networks and support vector machines.
method Proposed a method to establish the equivalence between infinitely wide neural networks trained by soft margin loss and standard soft margin SVMs with NTK trained by subgradient descent.
result The equivalence between NN and SVM is established, enabling practical applications such as non-vacuous generalization bounds and robustness certificates.
The paper proves a non-asymptotic test error approximation for KRR.
problem Understanding the test error of Kernel Ridge Regression.
method Established a non-asymptotic deterministic approximation for test error of KRR.
result The test error of KRR can be approximated by a closed-form estimate derived from the spectrum of the kernel operator.
Deep networks are mathematically equivalent to kernel machines learned by gradient descent.
problem Understanding the learned representations of deep learning models.
method Using gradient descent to learn deep networks, showing they are equivalent to kernel machines.
result Deep network weights are a superposition of training examples, revealing the learned function.
This work explores the connection between distances and kernels for conditional independence.
problem Measuring conditional independence in various fields like causal discovery and feature selection.
method Investigates the relationship between conditional independence measures induced by distances and reproducing kernels.
result Some kernel-based conditional independence measures are not equivalent to distance-based measures.
Paper proves non-equivalence of RKHS stability and kernel absolute summability.
problem Equivalence of RKHS stability and kernel absolute summability.
method Analyzes Reproducing Kernel Hilbert spaces and positive semidefinite kernels.
result Stable RKHSs can be induced by non-absolutely summable kernels.
Study deep maxout networks and their equivalence to Gaussian processes.
problem Understanding neural networks with infinite width.
method Derive equivalence between deep maxout networks and Gaussian processes, characterize maxout kernel, and provide efficient numerical implementation.
result Bayesian inference based on deep maxout network kernel leads to competitive results compared to finite-width counterparts and deep neural network kernels.
Study equivalence of metrics on noncompact Kähler manifolds with Bergman kernel properties.
problem Equivalence of invariant metrics on noncompact Kähler manifolds with bounded curvature.
method Use boundedness of ratio between Bergman kernel and wedge product of metric in fundamental domain.
result Equivalence of Bergman metric and Kähler-Einstein metric under specific conditions.
Clarifies connections between Nyström and SVGP methods for scalable GPs.
problem Lack of understanding between GP and kernel methods communities.
method Investigates Nyström and SVGP methods for scalable Gaussian processes.
result Establishes connections and equivalences between Nyström and SVGP methods.
The study shows inner-product kernels behave similarly to binary kernels in high dimensions.
problem Understanding the behavior of inner-product kernels in high-dimensional data.
method Investigation of eigenspectrum under binary mixture model using random matrix theory.
result The eigenspectrum of inner-product kernels is asymptotically equivalent to binary kernels.
Paper converts deep networks to flat, equivalent kernel machines.
problem Capacity control and uniform convergence in deep learning.
method Push-forward transformation from deep networks to indefinite kernel machines.
result Flat network weights are Lp-norm regularized (0<p<1).
Classifies vector fields in the kernel of a 1-form, up to equivalence.
problem Classifying vector fields in the kernel of a 1-form.
method Equivalence relation, local models, transversal unfoldings.
result Provides a list of local models and transversal unfoldings for vector fields.
Deep neural kernels and Laplace kernel have equivalent RKHS on spheres.
problem Comparing RKHS of deep neural tangent and Laplace kernels.
method Proof of RKHS equivalence using sphere restrictions and kernel properties.
result RKHS of deep neural tangent kernel and Laplace kernel are the same on Sd−1. CNNs become Gaussian processes with many filters, achieving state-of-the-art performance.
problem Training deep CNNs is computationally expensive.
method Showed that CNNs with many filters can be approximated by Gaussian processes, and computed the equivalent kernel efficiently.
result The kernel equivalent of a 32-layer ResNet achieves 0.84% classification error on MNIST.
We propose a vector-valued regression problem whose solution is equivalent to the reproducing kernel Hilbert space (RKHS) embedding of the Bayesian posterior distribution. This equivalence provides a new understanding of kernel Bayesian inference. Moreover, the optimization problem induces a new regularization for the …
The paper explores arbitrage opportunities in derivative markets under specific conditions.
problem Arbitrage opportunities in derivative markets under different conditions.
method Analyzes the relationship between pricing kernel monotonicity and stochastic arbitrage opportunities.
result Pricing kernel nonmonotonicity is equivalent to stochastic arbitrage opportunities under adequacy.
Study finds polynomial convergence rate for Farey sequences linked to Riemann hypothesis.
problem Understanding convergence rates of maximum mean discrepancies for Farey sequences.
method Identifying positive-semidefinite kernels and their polynomial convergence rates.
result Polynomial convergence rate of maximum mean discrepancies of Farey sequences is equivalent to the Riemann hypothesis.
A new method for deep Wishart processes improves kernel-based models.
problem Inference in deep Wishart processes is challenging due to the need for flexible distributions over positive semi-definite matrices.
method Developed a novel approach to flexible distributions over positive semi-definite matrices using the Bartlett decomposition of the Wishart probability density. Used this to create an approximate posterior for the DWP.
result Improved performance of inference in the DWP compared to DGP with equivalent prior.
The paper develops methods to handle missing data using regularized M-estimation in reproducing kernel Hilbert space.
problem Handling missing data in statistical analysis.
method Kernel ridge regression for imputation and maximum entropy method for propensity score estimation.
result The proposed methods achieve statistical consistency and asymptotic equivalence.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
We provide a unifying framework linking two classes of statistics used in two-sample and independence testing: on the one hand, the energy distances and distance covariances from the statistics literature; on the other, maximum mean discrepancies (MMD), that is, distances between embeddings of distributions to reproduc…
Study shows solutions of differential inclusions are homotopy equivalent in W1,p-topology.
problem Homotopy properties of solutions in differential inclusions.
method Analyzes differential inclusion with specific assumptions on corank one distribution.
result Solutions are homotopy equivalent to loop spaces in W1,p-topology. Unified derivation of high-dimensional linear models using stochastic gradient descent.
problem Performance analysis of high-dimensional linear models trained with stochastic gradient descent.
method Derivation of a deterministic equivalence for the two-point function of a random matrix resolvent.
result Unified understanding of model performance including previously known and novel results.
Inspired by Berndtsson's work on the subharmonicity property of the Bergman kernel, we give a local variation formula of the full Bergman kernels associated to deformations of complex manifolds. In compact case, it follows from the reproducing property of the Bergman kernel and the curvature formula of the 0-th direct …
The study explores how Matrix Product States can represent boolean and continuous functions.
problem Representing arbitrary boolean and continuous functions using Matrix Product States.
method Developed a construction method for MPS to represent boolean gates and proved density in continuous function space.
result MPS can accurately represent arbitrary boolean functions and continuous functions densely.
Graph-based kernels improve GP performance on graph data.
problem Improving Gaussian process performance on graph-structured data.
method Introduced graph neural network-inspired kernels into Gaussian processes.
result Graph convolutional networks are equivalent to certain GP kernels when infinitely wide.
In this paper, we study the behavior of Bergman kernels along the Kähler Ricci flow on Fano manifolds. We show that the Bergman kernels are equivalent along the Kähler Ricci flow under certain condition on the Ricci curvature of the initial metric. Then, using a recent work of Tian and Zhang, we can solve a conjecture …
New random feature maps for Laplacian and related kernels.
problem Challenges in approximating the Laplacian kernel and its generalizations.
method Developed random feature maps for Laplacian and related kernels, providing efficient sampling schemes.
result Demonstrated the efficacy of these random feature maps on real datasets.
The paper shows how multi-task learning in neural networks is similar to kernel regression and Hilbert spaces.
problem Understanding the solutions to multi-task shallow ReLU neural network learning problems.
method Analyzing the properties of solutions to multi-task shallow ReLU neural network learning problems, proving uniqueness and equivalence to minimum-norm interpolation problems in Hilbert spaces.
result The solutions to multi-task neural network interpolation problems are almost always unique and coincide with the solution to a minimum-norm interpolation problem in a Sobolev (Reproducing Kernel) Hilbert Space.
Eluder dimension and information gain are equivalent for reproducing kernel Hilbert spaces.
problem Complexity measures in bandit and reinforcement learning.
method Equivalence of eluder dimension and information gain for reproducing kernel Hilbert spaces.
result Eluder dimension and information gain are equivalent for reproducing kernel Hilbert spaces.
Deep neural networks and Gaussian processes are shown to be equivalent through activation functions.
problem Understanding the relationship between neural networks and Gaussian processes.
method Developing an equivalence theory based on activation functions and kernels.
result Models can be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased accuracy.
Develops noncommutative Cowen-Douglas theory for noncommuting operators.
problem Exploring noncommutative analogues of classical Cowen-Douglas theory.
method Defining noncommutative Cowen-Douglas class using matricial joint eigenvalues and showing equivalence classes are determined by associated noncommutative vector bundles.
result Unitary equivalence class of a tuple in the noncommutative Cowen-Douglas class is determined by the equivalence class of its associated noncommutative vector bundle.
We show that the mapping class group of a compact orientable surface with higher complexity has the following extreme rigidity in the sense of measure equivalence: if the mapping class group is measure equivalent to a discrete group, then they are commensurable up to finite kernel. Moreover, we describe all lattice emb…
New kernel model scales to large datasets.
problem Challenges in scaling kernel machines to large datasets.
method Introduced EigenPro 3.0, an algorithm based on projected dual preconditioned SGD.
result Ability to scale model and data sizes independently.
MKA incorporates manifold geometry into kernel alignment for more robust representation comparison.
problem Inadequate accounting for manifold geometry in kernel alignment metrics.
method Derives a theoretical framework for Manifold Approximated Kernel Alignment (MKA).
result MKA provides a more robust foundation for measuring representations.
We show that the logarithmic derivatives of the convolution heat kernels on a uni-modular Lie group are exponentially integrable. This result is then used to prove an "integrated" Harnack inequality for these heat kernels. It is shown that this integrated Harnack inequality is equivalent to a version of Wang's Harnack …
Study provides guarantees for kernel clustering under non-parametric mixtures.
problem Statistical guarantees for kernel-based clustering without strong assumptions.
method Non-parametric mixture models, kernel-based clustering, consistency guarantees.
result Necessary and sufficient separability conditions for consistent clustering recovery.