This paper analyzes convergence of FL for neural networks using NTK.
problem Theoretical guarantees of FL for neural networks with explicit forms and multi-step updates are unexplored.
method FL-NTK framework for federated learning of ReLU neural networks trained by gradient descent.
result FL-NTK converges to a global-optimal solution at a linear rate with proper learning parameters.
Generalizes NTK for surrogate gradient learning in neural networks.
problem Lack of theoretical foundation for surrogate gradient learning.
method Generalizes neural tangent kernel (NTK) for surrogate gradient learning (SGL).
result Surrogate gradient NTK provides a good characterization of SGL.
This paper improves neural network learning by escaping the NTK regime and efficiently learning sparse polynomials.
problem Learning sparse polynomials efficiently using neural networks.
method Spectral analysis of NTK, identifying 'good' directions, and constructing a regularizer.
result Gradient descent on a two-layer neural network can learn sparse polynomials efficiently, improving over the NTK and QuadNTK.
This paper explains robust overfitting in wide DNNs using adversarial training and NTK theory.
problem Robust overfitting in adversarially trained wide DNNs.
method Theoretical analysis using neural tangent kernel (NTK) theory and adversarial training dynamics.
result Adversarial training can lead to robust overfitting in wide DNNs, which can be mitigated by the proposed Adv-NTK method.
The paper tackles CF in CL by analyzing NTK overlap matrix and proposing OGD.
problem Catastrophic Forgetting in continual learning.
method Analysis of NTK overlap matrix, OGD with PCA.
result Proposes OGD to mitigate CF, supported by experiments.
Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to c…
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
problem Convergence analysis of averaged stochastic gradient descent for neural networks.
method Analyzed convergence of averaged stochastic gradient descent for overparameterized two-layer neural networks.
result Achieved minimax optimal convergence rate with global convergence guarantee.
ResNets and DenseNets converge to NTK with depth and width, offering advantages for kernel regression.
problem Understanding convergence of ResNets and DenseNets to Neural Tangent Kernel (NTK).
method Analysis of finite width and depth corrections for NTK of ResNets and DenseNets.
result ResNets and DenseNets can converge to NTK with depth and width, unlike vanilla networks.
Novel algorithms improve efficiency of finite width NTK computation.
problem Efficiency of computing finite width Neural Tangent Kernel (NTK).
method Leveraging neural network structure, propose two novel algorithms.
result Significantly improved efficiency in compute and memory requirements.
NTKs explain GNNs' alignment for graph prediction.
problem Understanding GNNs' alignment for graph prediction.
method Analyzing NTKs and alignment in GNNs, focusing on cross-covariance.
result Optimizing alignment in GNNs optimizes graph representation.
New framework establishes positivity of DNTK for PINNs.
problem Establishing positivity of NTK for PINNs with multiple differential operators.
method Proposed Differential Neural Tangent Kernel (DNTK) for PINNs.
result Positivity of infinite width DNTK for various activation functions and differential operators.
Deep learning dynamics and NTK evolution studied through diverse measures.
problem Understanding the training dynamics of deep neural networks and their loss landscapes.
method Phenomenological analysis of training dynamics in multiple architectures and datasets.
result Training dynamics exhibit a chaotic initial transient followed by a stable phase, with the NTK evolving to match full network performance.
New bounds on NTK's smallest eigenvalue for arbitrary data without distributional assumptions.
problem Existing bounds on NTK's smallest eigenvalue require distributional assumptions and high-dimensional data.
method Novel application of the hemisphere transform.
result Bounds on NTK's smallest eigenvalue hold with high probability even for constant input dimension.
Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…
The paper analyzes a neural network two-sample test using kernel analysis.
problem Determining if two datasets come from the same distribution.
method Time-analysis on a neural tangent kernel (NTK) two-sample test, extending to realistic neural network dynamics.
result Training times needed to detect deviations are well-separated in null and alternative hypothesis scenarios.
Gradient descent learns useful features even in the NTK regime.
problem The ability of neural networks to learn useful features.
method Local convergence analysis of gradient descent with regularization.
result Gradient descent can capture ground-truth directions for feature learning even after the loss threshold is reached.
This work analyzes PINNs for advection-diffusion equations using NTK theory.
problem Understanding and resolving the training difficulties of PINNs for advection-diffusion equations.
method Neural Tangent Kernel (NTK) analysis of PINNs for the linear advection-diffusion equation (LAD).
result PINNs struggle due to spectral bias and convergence rate disparity, especially in advection-dominated and diffusion-dominated regimes.
This work connects neural network training to convex optimization via the NTK.
problem Understanding and optimizing neural network training via convex programs.
method Interpreting gated ReLU network as MKL, showing NTK equivalence, and improving weights.
result The NTK cannot perform better than the optimal MKL kernel on the training set.
New stability bounds for GD in overparameterised shallow nets without NTK assumptions.
problem Generalisation and excess risk bounds for shallow neural networks.
method Oracle inequalities and stability analysis of GD without kernelisation.
result Oracle type bounds reveal GD's generalisation is controlled by an interpolating network with shortest GD path.
Random Fourier features improve tabular deep learning convergence.
problem Tabular deep learning convergence issues.
method Random Fourier projections as a pre-processing step, projecting inputs into a fixed feature space.
result Random Fourier pre-processing accelerates tabular deep learning convergence.
ResNets promote smoother interpolations than MLPs, enhancing generalization.
problem Understanding the difference in smoothness between ResNets and MLPs.
method Neural Tangent Kernel (NTK) analysis during gradient descent training.
result ResNet's NTK results in smoother interpolations than MLPs.
3-layer NTK models generalize better than 2-layer models, especially with large input dimensions.
problem Understanding the generalization of overparameterized neural networks.
method Analyzing the 3-layer NTK model's test error and comparing it to 2-layer NTK models.
result 3-layer NTK models have a faster descent in test error with respect to the number of neurons in the second hidden layer.
NTK-SAP improves neural network pruning by aligning training dynamics.
problem Improving neural network pruning to reduce training time and memory.
method Prune connections based on the spectrum of the Neural Tangent Kernel (NTK), using multiple random weight realizations and random inputs.
result Empirically, NTK-SAP achieves better performance than all baselines on multiple datasets.
Study of infinitely deep but narrow neural networks using NTK theory.
problem Analyzing the role of depth in deep learning with overparameterized networks.
method Infinite-depth limit analysis of MLP and CNN using Neural Tangent Kernel (NTK) theory.
result Established trainability guarantee for infinitely deep but narrow neural networks.
The evolution of a deep neural network trained by the gradient descent can be described by its neural tangent kernel (NTK) as introduced in [20], where it was proven that in the infinite width limit the NTK converges to an explicit limiting kernel and it stays constant during training. The NTK was also implicit in some…
The empirical NTK diverges from the NTK in classification problems during overtraining.
problem The divergence of empirical NTK from NTK in classification problems during overtraining.
method Demonstrated strictly positive definiteness of NTKs for FCNs and ResNets. Proved divergence of neural network parameters during training with cross-entropy loss.
result The empirical NTK does not uniformly converge to the NTK across all times on the training samples as the network width increases.
The paper provides approximation guarantees for neural networks trained with gradient flow.
problem Approximating neural networks trained with gradient flow in continuous L2(Sd−1)-norm. method NTK argument for non-convex second but last layer, under-parametrized regime.
result Gradient flow convergence guarantees for neural networks under Sobolev smoothness assumptions.
NTK neural networks are robust to adversarial attacks in nonparametric regression.
problem Adversarial robustness of neural networks in nonparametric regression.
method Gradient flow with early stopping for NTK neural networks, proving robustness in Sobolev spaces.
result NTK neural networks achieve optimal adversarial robustness rates in Sobolev spaces.
The study analyzes how many neurons are needed for two-layer neural networks trained with gradient descent.
problem Determining the minimum number of neurons required for effective training of shallow neural networks.
method Analyzes two-layer neural networks in the NTK regime, trained with gradient descent. Derives fast rates of convergence and tracks the number of hidden neurons required for generalization.
result Derives fast rates of convergence and improves on existing results for the number of hidden neurons needed for generalization.
The paper examines when NTK theory applies to real finite-width neural networks.
problem Understanding when NTK theory accurately predicts the behavior of finite-width neural networks.
method Empirical study of fully-connected ReLU and sigmoid DNNs with various hyperparameters and depths.
result NTK theory does not always apply to sufficiently deep networks with exploding gradients, and the kernel changes significantly during training.
The paper examines the neural tangent kernel for PINNs solving general PDEs and finds convergence conditions.
problem Analyzing the convergence of neural tangent kernel for PINNs solving general PDEs.
method Analysis of NTK initialization and convergence during training for general PDEs using PINNs.
result Homogeneity of differential operators is crucial for NTK convergence.
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
Generalizes neural tangent kernel analysis for two-layer networks with noise and regularization.
problem Limitations of NTK analysis in deep learning practice.
method Generalized NTK analysis for two-layer neural networks with weight decay and gradient noise.
result Noisy gradient descent with weight decay exhibits 'kernel-like' behavior and converges linearly.
The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectr…
This paper analyzes deep and wide transformer training dynamics.
problem Understanding the training dynamics of infinitely deep and wide transformers.
method Develops a mean-field framework for gradient-based training of transformers, controlling a neural PDE.
result Establishes a rigorous foundation for gradient-based transformer training, proving convergence to global minima.
Deep networks can classify data on smooth curves with high probability.
problem Classifying data from two disjoint smooth curves on the unit sphere.
method Gradient descent on a deep neural network, proving convergence and generalization via NTK dynamics.
result Randomly-initialized gradient descent learns to classify all points on the two curves with high probability when the network is sufficiently deep.
This paper explains GCNs using NTKs and improves their performance.
problem GCNs' performance degrades with depth, and skip connections marginally improve it.
method Derive NTKs for GCNs, validate with simulations, propose NTK as a surrogate model.
result Suitable normalisation can prevent drastic performance drop with depth.
Linearized attention fails to converge to NTK limit even at large widths.
problem Understanding the convergence of attention mechanisms to the kernel regime.
method Analyzes linearized attention and its relationship to the NTK limit, considering practical widths and conditions.
result Linearized attention does not converge to its NTK limit at any practical width, revealing a fundamental trade-off.
NTK theory fails to predict practical behavior of large-width neural networks.
problem Theoretical limits of NTK do not match practical neural network architectures.
method Empirical investigation of NTK's applicability to large-width architectures.
result Practically relevant behavior of large-width architectures differs from NTK theory.
A new method uses neural tangent kernel to efficiently compute MMD statistic.
problem Efficiently computing Maximum Mean Discrepancy (MMD) statistic with low memory and computational complexity.
method Identifies a connection between neural tangent kernel (NTK) and MMD to develop a computationally and memory-efficient approach.
result The proposed NTK-MMD statistic is validated through numerical experiments on synthetic and real-world datasets.
New NTK bounds show deep networks with minimum over-parameterization can still memorize and optimize.
problem Understanding memorization and optimization in sub-linear over-parameterized deep networks.
method Lower bound on NTK eigenvalues for deep networks with minimum over-parameterization.
result Deep networks with minimum over-parameterization can still be powerful memorizers and optimizers.
Improves understanding of neural network predictions using influence functions.
problem Challenges in understanding neural network predictions.
method Utilized NTK theory to calculate influence functions for over-parameterized neural networks.
result Proved that the approximation error of IF can be arbitrarily small in the over-parameterized regime.
Neural networks outperform NTK on compositional tasks, revealing a complexity gap.
problem Understanding the performance gap between neural networks and NTK on tasks with compositional structure.
method Characterized Fourier and architectural complexities, and analyzed the minimax rates of the architecture class.
result The NTK estimator is exponentially sub-optimal compared to the minimax floor when complexities decouple.
New algorithms use neural networks to minimize regret in contextual bandits.
problem Minimizing regret in sequential decisions with side information.
method Proposes NN-UCB and NTK-UCB algorithms using neural networks.
result Proves sublinear regret bounds for contextual bandits.
New approach to neural networks by incorporating observation noise and arbitrary prior means.
problem Misspecification on noisy data and limitations of NTK-GP equivalence.
method Introducing a regularizer for observation noise and proposing a shifted network for arbitrary prior means.
result Removes key obstacles to practical Gaussian process modeling in neural networks.
Kempe discusses NTK approach to machine learning problems.
problem Generally unsolvable machine learning problems.
method NTK approach focusing on kernel formulations.
result Practical applications like data distillation and adversarial robustness.
DEQs and explicit networks are nearly equivalent for Gaussian mixtures.
problem Understanding the equivalence between DEQs and explicit neural networks.
method Random matrix theory and analysis of kernel matrices.
result A shallow explicit network can mimic the kernel of a DEQ.
Recent theoretical work has established connections between over-parametrized neural networks and linearized models governed by he Neural Tangent Kernels (NTKs). NTK theory leads to concrete convergence and generalization results, yet the empirical performance of neural networks are observed to exceed their linearized …