The neural tangent kernel equivalence theorem fails in practice.
problem Does the neural tangent kernel (NTK) equivalence theorem hold in practical neural network training?
method Rigorously derived NTK and conducted numerical experiments to evaluate the equivalence theorem.
result Adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in predictor error.
The paper examines the neural tangent kernel for PINNs solving general PDEs and finds convergence conditions.
problem Analyzing the convergence of neural tangent kernel for PINNs solving general PDEs.
method Analysis of NTK initialization and convergence during training for general PDEs using PINNs.
result Homogeneity of differential operators is crucial for NTK convergence.
A new method uses neural tangent kernel to efficiently compute MMD statistic.
problem Efficiently computing Maximum Mean Discrepancy (MMD) statistic with low memory and computational complexity.
method Identifies a connection between neural tangent kernel (NTK) and MMD to develop a computationally and memory-efficient approach.
result The proposed NTK-MMD statistic is validated through numerical experiments on synthetic and real-world datasets.
Improved standard parameterization yields well-defined neural tangent kernel.
problem Extrapolation of standard parameterization to infinite width is problematic.
method Proposed an improved extrapolation of the standard parameterization.
result Improved standard parameterization yields similar accuracy to NTK parameterization but with better correspondence to finite width networks.
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
problem Convergence analysis of averaged stochastic gradient descent for neural networks.
method Analyzed convergence of averaged stochastic gradient descent for overparameterized two-layer neural networks.
result Achieved minimax optimal convergence rate with global convergence guarantee.
This paper analyzes convergence of FL for neural networks using NTK.
problem Theoretical guarantees of FL for neural networks with explicit forms and multi-step updates are unexplored.
method FL-NTK framework for federated learning of ReLU neural networks trained by gradient descent.
result FL-NTK converges to a global-optimal solution at a linear rate with proper learning parameters.
The evolution of a deep neural network trained by the gradient descent can be described by its neural tangent kernel (NTK) as introduced in [20], where it was proven that in the infinite width limit the NTK converges to an explicit limiting kernel and it stays constant during training. The NTK was also implicit in some…
Empirical study shows standard CNNs deviate from NTK predictions.
problem Understanding how standard finite-width CNNs behave compared to their infinite-width NTK counterparts.
method Empirical analysis of AlexNet and LeNet architectures.
result Standard CNNs deviate significantly from their NTK counterparts, but deviation decreases with wider networks.
The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.
problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.
method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.
result Obtained an approximation formula for functions represented by 2-layer neural networks.
The paper examines when NTK theory applies to real finite-width neural networks.
problem Understanding when NTK theory accurately predicts the behavior of finite-width neural networks.
method Empirical study of fully-connected ReLU and sigmoid DNNs with various hyperparameters and depths.
result NTK theory does not always apply to sufficiently deep networks with exploding gradients, and the kernel changes significantly during training.
Novel algorithms improve efficiency of finite width NTK computation.
problem Efficiency of computing finite width Neural Tangent Kernel (NTK).
method Leveraging neural network structure, propose two novel algorithms.
result Significantly improved efficiency in compute and memory requirements.
New framework establishes positivity of DNTK for PINNs.
problem Establishing positivity of NTK for PINNs with multiple differential operators.
method Proposed Differential Neural Tangent Kernel (DNTK) for PINNs.
result Positivity of infinite width DNTK for various activation functions and differential operators.
The empirical NTK diverges from the NTK in classification problems during overtraining.
problem The divergence of empirical NTK from NTK in classification problems during overtraining.
method Demonstrated strictly positive definiteness of NTKs for FCNs and ResNets. Proved divergence of neural network parameters during training with cross-entropy loss.
result The empirical NTK does not uniformly converge to the NTK across all times on the training samples as the network width increases.
Generalizes NTK for surrogate gradient learning in neural networks.
problem Lack of theoretical foundation for surrogate gradient learning.
method Generalizes neural tangent kernel (NTK) for surrogate gradient learning (SGL).
result Surrogate gradient NTK provides a good characterization of SGL.
NTK-SAP improves neural network pruning by aligning training dynamics.
problem Improving neural network pruning to reduce training time and memory.
method Prune connections based on the spectrum of the Neural Tangent Kernel (NTK), using multiple random weight realizations and random inputs.
result Empirically, NTK-SAP achieves better performance than all baselines on multiple datasets.
This paper improves neural tangent kernels for better generalization and local elasticity.
problem Performance gap between neural tangent kernels and real-world neural networks.
method Introduces label-aware kernels using Hoeffding decomposition.
result Models trained with proposed kernels simulate NNs better in terms of generalization and local elasticity.
Laplace kernel and Neural Tangent Kernels are shown to be nearly identical for normalized data.
problem Understanding the similarity between Laplace and Neural Tangent Kernels.
method Theoretical analysis and experiments on normalized data.
result Laplace kernel and Neural Tangent Kernels have nearly identical eigenfunctions and RKHS for normalized data.
This paper explains GCNs using NTKs and improves their performance.
problem GCNs' performance degrades with depth, and skip connections marginally improve it.
method Derive NTKs for GCNs, validate with simulations, propose NTK as a surrogate model.
result Suitable normalisation can prevent drastic performance drop with depth.
Generalizes neural tangent kernel analysis for two-layer networks with noise and regularization.
problem Limitations of NTK analysis in deep learning practice.
method Generalized NTK analysis for two-layer neural networks with weight decay and gradient noise.
result Noisy gradient descent with weight decay exhibits 'kernel-like' behavior and converges linearly.
New kernels from neural networks show better performance than traditional methods.
problem Improving neural network performance on small datasets.
method Developed algebraic operations to create compositional kernels from neural network architectures.
result Compositional kernels achieve higher accuracy than neural tangent kernels and neural networks on small datasets.
Efficiently applies NTK to large-scale datasets using random features.
problem Computational limitations of kernel methods for large-scale datasets.
method Proposes a sketching-based algorithm combining random features of arc-cosine kernels to construct an efficient feature map of the NTK.
result Achieves comparable error bounds to exact kernel methods but with significantly reduced feature dimensionality.
NTK theory fails to predict practical behavior of large-width neural networks.
problem Theoretical limits of NTK do not match practical neural network architectures.
method Empirical investigation of NTK's applicability to large-width architectures.
result Practically relevant behavior of large-width architectures differs from NTK theory.
ResNets and DenseNets converge to NTK with depth and width, offering advantages for kernel regression.
problem Understanding convergence of ResNets and DenseNets to Neural Tangent Kernel (NTK).
method Analysis of finite width and depth corrections for NTK of ResNets and DenseNets.
result ResNets and DenseNets can converge to NTK with depth and width, unlike vanilla networks.
Study on Neural Tangent Kernel of Matrix Product States and their convergence.
problem Understanding the convergence of Neural Tangent Kernel of Matrix Product States.
method Analyzing the Neural Tangent Kernel of Matrix Product States and proving its convergence in the infinite bond dimensional limit.
result The Neural Tangent Kernel of Matrix Product States converges to a constant matrix during training.
Tensor programs prove neural network limits for any architecture.
problem Understanding the limits of neural networks of any architecture.
method Prove convergence of neural network's Tangent Kernel (NTK) to a deterministic limit as network widths increase.
result Identify conditions for correct NTK limit calculation based on gradient independence assumption.
Neural networks outperform NTK on compositional tasks, revealing a complexity gap.
problem Understanding the performance gap between neural networks and NTK on tasks with compositional structure.
method Characterized Fourier and architectural complexities, and analyzed the minimax rates of the architecture class.
result The NTK estimator is exponentially sub-optimal compared to the minimax floor when complexities decouple.
NTKs explain GNNs' alignment for graph prediction.
problem Understanding GNNs' alignment for graph prediction.
method Analyzing NTKs and alignment in GNNs, focusing on cross-covariance.
result Optimizing alignment in GNNs optimizes graph representation.
Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…
Study on how adversarial training affects neural network kernels and robustness.
problem Understanding and improving adversarial robustness in neural networks.
method Empirical study of the evolution of the empirical Neural Tangent Kernel (NTK) under standard and adversarial training.
result Adversarial training leads to a new kernel that provides robustness, even when non-robust training is performed on top of it.
Paper introduces RNTK for recurrent neural networks, improving performance across various datasets.
problem Understanding and optimizing overparametrized recurrent neural networks.
method Developed the Recurrent Neural Tangent Kernel (RNTK) to compare inputs of different lengths.
result RNTK offers significant performance gains over other kernels, including standard NTKs, across multiple datasets.
Paper calculates eigenvalue decay rates for neural network kernels on general domains.
problem Determining eigenvalue decay rates for neural network kernels on arbitrary domains.
method Proved dynamics of wide neural networks approximates NTK on general domains, used minimax optimality and interpolation spaces.
result Provided strategy to calculate eigenvalue decay rates for neural network kernels.
The study explains how neural networks align their kernels to target functions during training.
problem Understanding how neural networks align their kernels to target functions during training.
method Theoretical analysis of kernel evolution in toy models and deep networks.
result Kernel alignment naturally emerges during training to accelerate convergence and improve generalization.
Paper analyzes soft tree ensembles using NTK, finding only leaf count matters.
problem Understanding impact of various tree architectures in ensemble learning.
method Formulated and analyzed Neural Tangent Kernel (NTK) for soft tree ensembles.
result Only the number of leaves at each depth is relevant for tree architecture in ensemble learning.
This work connects neural network training to convex optimization via the NTK.
problem Understanding and optimizing neural network training via convex programs.
method Interpreting gated ReLU network as MKL, showing NTK equivalence, and improving weights.
result The NTK cannot perform better than the optimal MKL kernel on the training set.
Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to c…
New bounds on NTK's smallest eigenvalue for arbitrary data without distributional assumptions.
problem Existing bounds on NTK's smallest eigenvalue require distributional assumptions and high-dimensional data.
method Novel application of the hemisphere transform.
result Bounds on NTK's smallest eigenvalue hold with high probability even for constant input dimension.
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network functio…
PINNs fail to train due to NTK convergence rate discrepancies.
problem Understanding why PINNs fail to train during gradient descent.
method Analyzing PINNs through the Neural Tangent Kernel (NTK) perspective.
result PINNs' NTK converges to a deterministic kernel with constant convergence rate during training.
GNTK reveals convergence of GNNs on large graphs.
problem Understanding and optimizing GNNs on large graphs.
method Graph Neural Tangent Kernels (GNTK) and graphons.
result GNTKs converge to graphon NTKs on large graphs, enabling task inference.
This paper tightens bounds on the smallest eigenvalue of NTK for deep ReLU networks.
problem Analyzing the smallest eigenvalue of Neural Tangent Kernel for deep ReLU networks.
method Analyzing various quantities of independent interest, including lower bounds on the smallest singular value of hidden feature matrices and upper bounds on the Lipschitz constant of input-output feature maps.
result Tight bounds on the smallest eigenvalue of NTK matrices for deep ReLU nets, both in the limiting case of infinite widths and for finite widths.
Study of infinitely deep but narrow neural networks using NTK theory.
problem Analyzing the role of depth in deep learning with overparameterized networks.
method Infinite-depth limit analysis of MLP and CNN using Neural Tangent Kernel (NTK) theory.
result Established trainability guarantee for infinitely deep but narrow neural networks.
Deep learning dynamics and NTK evolution studied through diverse measures.
problem Understanding the training dynamics of deep neural networks and their loss landscapes.
method Phenomenological analysis of training dynamics in multiple architectures and datasets.
result Training dynamics exhibit a chaotic initial transient followed by a stable phase, with the NTK evolving to match full network performance.
New framework for understanding infinite-width neural networks.
problem Understanding the infinite-width limit behavior of neural networks.
method General framework to study limit behavior of neural models based on hyperparameter scaling.
result Derives scaling for existing mean-field and neural tangent kernel limits and introduces new dynamically stable limits.
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
Orthogonal initialization does not speed up training in ultra-wide neural networks.
problem Exploring the effect of orthogonal initialization on training speed in deep neural networks.
method Study of neural tangent kernel dynamics in FCNs and CNNs with orthogonal initialization.
result The NTK of orthogonally-initialized networks remains constant during training, suggesting no speedup in the NTK regime.
ResNets promote smoother interpolations than MLPs, enhancing generalization.
problem Understanding the difference in smoothness between ResNets and MLPs.
method Neural Tangent Kernel (NTK) analysis during gradient descent training.
result ResNet's NTK results in smoother interpolations than MLPs.
This paper examines the impact of random initialization in neural networks using NTK theory.
problem Understanding the impact of random initialization in neural networks using NTK theory.
method Analyzes the convergence of training dynamics and generalization error of wide neural networks with random initialization.
result The generalization error of wide neural networks trained by gradient descent is \( \Omega(n^{-\frac{3}{d+3}}) \), highlighting the benefits of mirror initialization and suggesting limitations of NTK theory.
This paper explains robust overfitting in wide DNNs using adversarial training and NTK theory.
problem Robust overfitting in adversarially trained wide DNNs.
method Theoretical analysis using neural tangent kernel (NTK) theory and adversarial training dynamics.
result Adversarial training can lead to robust overfitting in wide DNNs, which can be mitigated by the proposed Adv-NTK method.