This paper explains why ResNets generalize better than FFNets using neural tangent kernels.
problem Understanding why deep ResNets generalize better than deep FFNets.
method Using neural tangent kernels to compare the learnability of functions induced by the kernels of ResNets and FFNets.
result The kernel of ResNets does not exhibit degeneracy as depth increases, unlike FFNets.
Deep neural kernels and Laplace kernel have equivalent RKHS on spheres.
problem Comparing RKHS of deep neural tangent and Laplace kernels.
method Proof of RKHS equivalence using sphere restrictions and kernel properties.
result RKHS of deep neural tangent kernel and Laplace kernel are the same on Sd−1. Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task lea…
Deep kernel processes unify various models using Gram matrices and kernel functions.
problem Unified representation of various deep learning models.
method Defining deep kernel processes with progressively transformed Gram matrices and sampling from inverse Wishart distributions.
result Deep Gaussian processes, BNNs, infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes.
Deep ReLU networks approximate as well as shallow ones in kernel regimes.
problem Understanding the limitations of kernel methods for deep ReLU networks.
method Characterizing eigenvalue decays of kernels derived from deep ReLU networks.
result Deep ReLU networks and shallow two-layer networks have equivalent approximation properties in kernel regimes.
We introduce scalable deep kernels, which combine the structural properties of deep learning architectures with the non-parametric flexibility of kernel methods. Specifically, we transform the inputs of a spectral mixture base kernel with a deep architecture, using local kernel interpolation, inducing points, and struc…
Deep neural nets optimize kernel parameters for non-parametric two-sample tests.
problem Determining if two samples come from the same distribution.
method Deep kernels trained to maximize test power, adapting to distribution smoothness and shape.
result Deep kernels outperform simpler kernels in high dimensions and complex data.
Improved deep kernel machine achieves 94.5% test accuracy on CIFAR-10.
problem Achieving high accuracy on complex tasks like image classification.
method Stochastic kernel regularisation to add noise to learned Gram matrices.
result 94.5% test accuracy on CIFAR-10.
Deep kernel learning provides an elegant and principled framework for combining the structural properties of deep learning algorithms with the flexibility of kernel methods. By means of a deep neural network, we learn a parametrized kernel operator that can be combined with a differentiable kernel algorithm during infe…
A new method for deep Wishart processes improves kernel-based models.
problem Inference in deep Wishart processes is challenging due to the need for flexible distributions over positive semi-definite matrices.
method Developed a novel approach to flexible distributions over positive semi-definite matrices using the Bartlett decomposition of the Wishart probability density. Used this to create an approximate posterior for the DWP.
result Improved performance of inference in the DWP compared to DGP with equivalent prior.
Deep learning framework for kernel methods using RKHM and Perron-Frobenius operators.
problem Kernel methods in deep learning with potential overfitting issues.
method Combining RKHM and Perron-Frobenius operator to derive a new Rademacher bound and analyze deep kernel methods.
result Theoretical interpretation of benign overfitting and milder dependency on output dimension.
Study deep maxout networks and their equivalence to Gaussian processes.
problem Understanding neural networks with infinite width.
method Derive equivalence between deep maxout networks and Gaussian processes, characterize maxout kernel, and provide efficient numerical implementation.
result Bayesian inference based on deep maxout network kernel leads to competitive results compared to finite-width counterparts and deep neural network kernels.
A new model DKMPP integrates covariates and uses an integration-free method for spatio-temporal point processes.
problem Training intractable deep spatio-temporal point processes with multimodal covariates.
method DKMPP uses a deep kernel to model complex relationships and an integration-free score matching method.
result DKMPP and score-based estimators outperform baseline models in spatio-temporal point processes.
In this paper, we propose a novel supervised learning method that is called Deep Embedding Kernel (DEK). DEK combines the advantages of deep learning and kernel methods in a unified framework. More specifically, DEK is a learnable kernel represented by a newly designed deep architecture. Compared with pre-defined kerne…
New kernel connects deep learning to optimization, improving generalization.
problem Understanding why deep neural networks generalize well at over-parameterization.
method Established Neural Optimization Kernel (NOK) linking deep NN to optimization problems.
result New generalization bound for deep structured approximated NOK architecture.
DHGAK aligns substructures for better graph kernel performance.
problem Limited performance of traditional graph kernels due to missing substructure similarities.
method Hierarchically aligns relational substructures in deep embedding space, assigning same feature maps in RKHS.
result DHGAK outperforms state-of-the-art graph kernels on various benchmarks.
Physics Informed Deep Kernel Learning improves prediction accuracy and uncertainty quantification.
problem Limited performance of deep kernel learning due to scarce or insufficient data.
method Integrates physics knowledge represented by differential equations with latent sources into deep kernel learning.
result Advantages in prediction accuracy and uncertainty quantification on synthetic and real-world datasets.
Deep networks are shown to be equivalent to a new type of kernel chain.
problem Identifying an appropriate function space for deep neural networks.
method Extending Reproducing Kernel Banach Spaces (RKBS) to chain RKBS (cRKBS), which composes kernels rather than functions.
result Any deep neural network function is a neural cRKBS function, and conversely, any neural cRKBS function corresponds to a deep neural network.
A new method combines deep kernels with Gaussian processes to avoid overfitting.
problem Losing Bayesian benefits in deep kernel learning due to kernel optimization.
method Using Infinite-width neural networks and Neural Network Gaussian Process (NNGP) as a guide for DKL optimization.
result Robustness to overfitting and good predictive performance on various datasets.
DKL-KAN combines deep learning and kernel methods for scalable, expressive models.
problem Combining deep learning's depth with kernel methods' flexibility for scalable models.
method DKL-KAN uses Kolmogorov-Arnold Networks (KAN) to optimize kernel attributes within a Gaussian process framework.
result DKL-KAN outperforms DKL-MLP on datasets with a low number of observations and DKL-MLP on large datasets.
Deep networks are mathematically equivalent to kernel machines learned by gradient descent.
problem Understanding the learned representations of deep learning models.
method Using gradient descent to learn deep networks, showing they are equivalent to kernel machines.
result Deep network weights are a superposition of training examples, revealing the learned function.
Inspired by a growing interest in analyzing network data, we study the problem of node classification on graphs, focusing on approaches based on kernel machines. Conventionally, kernel machines are linear classifiers in the implicit feature space. We argue that linear classification in the feature space of kernels comm…
Kernel learning methods are among the most effective learning methods and have been vigorously studied in the past decades. However, when tackling with complicated tasks, classical kernel methods are not flexible or "rich" enough to describe the data and hence could not yield satisfactory performance. In this paper, vi…
This work introduces a new quantum kernel, quantum tangent kernel, for improved performance.
problem Improving quantum machine learning performance beyond conventional methods.
method Developed a deep parameterized quantum circuit and used first-order expansion for training.
result The quantum tangent kernel outperforms conventional quantum kernel methods for ansatz-generated datasets.
Paper converts deep networks to flat, equivalent kernel machines.
problem Capacity control and uniform convergence in deep learning.
method Push-forward transformation from deep networks to indefinite kernel machines.
result Flat network weights are Lp-norm regularized (0<p<1).
Proposes DAK model for improved GP computations.
problem Challenges in high-dimensional GP layers in DKL.
method Additive structure and induced prior approximation for GP units.
result Outperforms state-of-the-art DKL methods in regression and classification.
A new model combines deep learning and Gaussian Processes with hyperdata learning.
problem Combining deep learning and Gaussian Processes for expressive and robust learning.
method Conditional Deep Gaussian Process (DGP) with hyperdata learning and approximate inference.
result Conditional DGP offers better expressiveness and robustness compared to existing methods.
We investigate iterated compositions of weighted sums of Gaussian kernels and provide an interpretation of the construction that shows some similarities with the architectures of deep neural networks. On the theoretical side, we show that these kernels are universal and that SVMs using these kernels are universally con…
Optimal kernel in KR can be data-dependent, improving model performance.
problem Fixed kernel in KR limits model performance.
method Considered data-dependent kernels for KR, using posterior covariance.
result Data-dependent kernel choice leads to optimal performance.
Deep Gaussian Processes with polynomial kernels can collapse rapidly without proper hyperparameter tuning.
problem The collapse of Deep Gaussian Processes with polynomial kernels without careful hyperparameter tuning.
method Analysis using the Berry-Esseen Theorem and observation of prior behavior.
result The prior of a Deep Gaussian Process collapses rapidly towards zero or places negligible mass on low norm functions without proper hyperparameter tuning.
Learning with kernels is an important concept in machine learning. Standard approaches for kernel methods often use predefined kernels that require careful selection of hyperparameters. To mitigate this burden, we propose in this paper a framework to construct and learn a data-dependent kernel based on random features …
Advances in deep learning for spatio-temporal event modeling.
problem Limitations of traditional parametric models in capturing nonstationary dynamics.
method Integration of deep neural architectures to model conditional intensity function and influence kernels.
result Deep influence kernel approach enhances expressiveness and statistical explainability.
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.
Building highly non-linear and non-parametric models is central to several state-of-the-art machine learning systems. Kernel methods form an important class of techniques that induce a reproducing kernel Hilbert space (RKHS) for inferring non-linear models through the construction of similarity functions from data. The…
Automates kernel discovery for longitudinal data analysis.
problem Handling irregularly sampled, sparse longitudinal data with multilevel correlation.
method Combines deep neural networks and non-parametric kernel methods to discover complex multilevel correlation structure.
result Significantly outperforms state-of-the-art methods on benchmark data sets.
Deep neural networks for structured prediction using kernel-induced losses.
problem Structured prediction tasks for images and texts.
method Designing a novel family of deep neural architectures that predict in a finite-dimensional subspace derived from the kernel-induced loss.
result Gradient descent algorithms can be used for structured prediction with deep neural networks.
Deep neural-kernel models combine neural networks and kernel machines for scalable large datasets.
problem Combining neural networks and kernel machines for efficient large-scale learning.
method Hybrid neural-kernel architecture using explicit feature mapping and pooling layers.
result The deep neural-kernel models are effective and scalable on benchmark datasets.
Study on deep neural networks using branching processes and Mehler's formula.
problem Understanding the mathematical role of activation functions in compositional neural networks.
method Connection between compositional kernels and branching processes via Mehler's formula; new random features algorithm.
result Explicit formulas for eigenvalues of compositional kernels quantify complexity.
We apply a fast kernel method for mask-based single-channel speech enhancement. Specifically, our method solves a kernel regression problem associated to a non-smooth kernel function (exponential power kernel) with a highly efficient iterative method (EigenPro). Due to the simplicity of this method, its hyper-parameter…
New kernels from ELU and GELU networks reveal non-trivial fixed points.
problem Understanding fixed-point dynamics in deep neural networks with ELU and GELU activations.
method Deriving covariance functions and analyzing fixed-point dynamics of ELU and GELU networks.
result ELU and GELU networks exhibit non-trivial fixed-point dynamics, explaining implicit regularization in overparameterized models.
Convolutional DKMs improve kernel methods on MNIST, CIFAR-10, and CIFAR-100.
problem Improving kernel methods for image classification.
method Developed a novel inter-domain inducing point approximation and introduced various techniques to extend DKMs to convolutional networks.
result Achieved state-of-the-art performance on image classification benchmarks.
Develops a deep non-stationary kernel for non-stationary spatio-temporal point processes.
problem Capturing non-stationary dependencies in point process data.
method Approximates the influence kernel with a novel low-rank decomposition and introduces a log-barrier penalty to maintain non-negativity.
result Demonstrates superior performance and computational efficiency compared to state-of-the-art methods.
Survival kernets scale deep kernel survival analysis to large datasets with interpretability and theoretical guarantees.
problem Scalable and interpretable deep kernel survival analysis for large datasets.
method Survival kernets use kernel netting for training set compression and XGBoost for warm-starting neural architecture search.
result Survival kernets achieve optimal time-dependent concordance index on various survival analysis datasets.
The paper tackles counterfactual inference with multioutput deep kernels in high-dimensional settings.
problem Performing counterfactual inference with observational data in high-dimensional settings with multiple actions and outcomes.
method The paper presents a general class of counterfactual multi-task deep kernels models based on Structural Causal Models (SCM) and Gaussian Processes.
result The models estimate causal effects and learn policies efficiently, scaling well with high dimensions.
Deep Gaussian Processes are reinterpreted as deep trigonometric networks for tractable inference.
problem Challenging inference in DGPs due to intractable marginalization in latent function space.
method Viewing DGPs as deep trigonometric networks with Bochner's theorem, and using the wide limit with a bottleneck to translate DGPs into deep trigonometric networks.
result The weight space view yields the same effective covariance functions as obtained in function space, and varying prior distributions over network parameters is equivalent to employing different kernels.
Deep convolutional networks can be understood through kernel methods, providing insights into their inductive bias.
problem Understanding the functional space and inductive bias of deep convolutional networks.
method Using kernel methods to analyze simple hierarchical kernels with convolution and pooling layers.
result The RKHS consists of additive models of interaction terms between patches, and pooling layers encourage spatial similarities.
Generalization performance of classifiers in deep learning has recently become a subject of intense study. Deep models, typically over-parametrized, tend to fit the training data exactly. Despite this "overfitting", they perform well on test data, a phenomenon not yet fully understood. The first point of our paper is t…
A novel GP architecture, Thin and Deep GP, learns lower-dimensional representations without losing interpretability.
problem Challenges in selecting appropriate kernel for Gaussian processes.
method Proposes a novel synthesis of deep and shallow GP approaches, parameterizing lengthscale in a way that maintains interpretability and learns lower-dimensional embeddings.
result TDGP discovers lower-dimensional manifolds in input data, performs well in benchmark datasets, and behaves well with increasing layers.