The paper tackles CF in CL by analyzing NTK overlap matrix and proposing OGD.
problem Catastrophic Forgetting in continual learning.
method Analysis of NTK overlap matrix, OGD with PCA.
result Proposes OGD to mitigate CF, supported by experiments.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “NTK Overlap Matrix”
The empirical NTK diverges from the NTK in classification problems during overtraining.
problem The divergence of empirical NTK from NTK in classification problems during overtraining.
method Demonstrated strictly positive definiteness of NTKs for FCNs and ResNets. Proved divergence of neural network parameters during training with cross-entropy loss.
result The empirical NTK does not uniformly converge to the NTK across all times on the training samples as the network width increases.
Study on Neural Tangent Kernel of Matrix Product States and their convergence.
problem Understanding the convergence of Neural Tangent Kernel of Matrix Product States.
method Analyzing the Neural Tangent Kernel of Matrix Product States and proving its convergence in the infinite bond dimensional limit.
result The Neural Tangent Kernel of Matrix Product States converges to a constant matrix during training.
Study on overlaps of singular vectors in Gaussian matrix submatrices.
problem Analyzing overlaps of singular vectors in submatrices of Gaussian matrices.
method Utilizes dynamics of singular vectors and specific resolvents for Brownian trajectories.
result Explicit forms for limiting rescaled mean squared overlaps in the bulk of spectra.
Study on Langevin dynamics for recovering planted signals in spiked matrix models.
problem Recovering a planted signal in spiked matrix models.
method Path-wise characterization of overlap using integro-differential equations and explicit formula derivation.
result Sharp phase transition in limiting overlap: positive in one regime, zero in another due to injected noise.
Linearized attention fails to converge to NTK limit even at large widths.
problem Understanding the convergence of attention mechanisms to the kernel regime.
method Analyzes linearized attention and its relationship to the NTK limit, considering practical widths and conditions.
result Linearized attention does not converge to its NTK limit at any practical width, revealing a fundamental trade-off.
NTKs explain GNNs' alignment for graph prediction.
problem Understanding GNNs' alignment for graph prediction.
method Analyzing NTKs and alignment in GNNs, focusing on cross-covariance.
result Optimizing alignment in GNNs optimizes graph representation.
Tensor programs prove neural network limits for any architecture.
problem Understanding the limits of neural networks of any architecture.
method Prove convergence of neural network's Tangent Kernel (NTK) to a deterministic limit as network widths increase.
result Identify conditions for correct NTK limit calculation based on gradient independence assumption.
"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approachstat.ML
Lossless compression of deep neural networks using NTK and RMT.
problem Compressing large-scale deep neural networks for low-power devices.
method High-dimensional neural tangent kernel approach.
result Asymptotic spectral equivalence between NTK matrices of wide DNNs enables lossless compression.
DEQs and explicit networks are nearly equivalent for Gaussian mixtures.
problem Understanding the equivalence between DEQs and explicit neural networks.
method Random matrix theory and analysis of kernel matrices.
result A shallow explicit network can mimic the kernel of a DEQ.
GAME improves matrix completion by considering subgroup-specific latent structures.
problem Heterogeneous data with overlapping categories, smoothing away subgroup-specific variation.
method Group-Aware Matrix Estimation (GAME) with overlapping nuclear-norm penalties.
result GAME outperforms global low-rank estimators in structured missingness regimes.
Efficiently applies NTK to large-scale datasets using random features.
problem Computational limitations of kernel methods for large-scale datasets.
method Proposes a sketching-based algorithm combining random features of arc-cosine kernels to construct an efficient feature map of the NTK.
result Achieves comparable error bounds to exact kernel methods but with significantly reduced feature dimensionality.
Producing overlapping schemes is a major issue in clustering. Recent proposed overlapping methods relies on the search of an optimal covering and are based on different metrics, such as Euclidean distance and I-Divergence, used to measure closeness between observations. In this paper, we propose the use of another meas…
Novel algorithm speeds up log-determinant estimation for large matrices.
problem Efficiently estimating log-determinants of large positive definite matrices under memory constraints.
method Hierarchical algorithm based on block-wise computation of LDL decomposition.
result Accurate estimation of NTK log-determinants from a tiny fraction of the full dataset.
Study eigenvector overlaps in large Gaussian matrices, simplifying for GOE.
problem Investigate eigenvector overlaps in large Gaussian matrices.
method Analysis of eigenvector flow under Dyson Brownian motion.
result Explicit computation of limiting rescaled mean squared overlaps.
On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscs.LG
Gradient descent proves global convergence for deep networks with a single wide layer.
problem Proving global convergence of gradient descent for deep ReLU networks.
method Simplified proof using a single wide layer, leveraging ReLU's Lipschitz property.
result Gradient descent converges globally for networks with a single wide layer.
3-layer NTK models generalize better than 2-layer models, especially with large input dimensions.
problem Understanding the generalization of overparameterized neural networks.
method Analyzing the 3-layer NTK model's test error and comparing it to 2-layer NTK models.
result 3-layer NTK models have a faster descent in test error with respect to the number of neurons in the second hidden layer.
NTK-SAP improves neural network pruning by aligning training dynamics.
problem Improving neural network pruning to reduce training time and memory.
method Prune connections based on the spectrum of the Neural Tangent Kernel (NTK), using multiple random weight realizations and random inputs.
result Empirically, NTK-SAP achieves better performance than all baselines on multiple datasets.
On the overlaps between eigenvectors of correlated random matricescond-mat.stat-mech
We obtain general, exact formulas for the overlaps between the eigenvectors of large correlated random matrices, with additive or multiplicative noise. These results have potential applications in many different contexts, from quantum thermalisation to high dimensional statistics. We find that the overlaps only depend …
This paper presents a novel spectral algorithm with additive clustering designed to identify overlapping communities in networks. The algorithm is based on geometric properties of the spectrum of the expected adjacency matrix in a random graph model that we call stochastic blockmodel with overlap (SBMO). An adaptive ve…
The evolution of a deep neural network trained by the gradient descent can be described by its neural tangent kernel (NTK) as introduced in [20], where it was proven that in the infinite width limit the NTK converges to an explicit limiting kernel and it stays constant during training. The NTK was also implicit in some…
Paper proposes a new co-clustering method for overlapping clusters and outliers.
problem Real-world datasets often contain overlaps and outliers in co-clusters.
method Formulated Non-Exhaustive, Overlapping Co-Clustering problem and developed NEO-CC algorithm.
result NEO-CC algorithm effectively captures underlying co-clustering structure of real-world data.
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
NTK neural networks are robust to adversarial attacks in nonparametric regression.
problem Adversarial robustness of neural networks in nonparametric regression.
method Gradient flow with early stopping for NTK neural networks, proving robustness in Sobolev spaces.
result NTK neural networks achieve optimal adversarial robustness rates in Sobolev spaces.
A Theoretical Framework for LLM Fine-tuning Using Early Stopping for Non-random Initializationstat.ML
Theoretical framework explains why few epochs are enough for LLM fine-tuning.
problem Understanding why few epochs are sufficient for LLM fine-tuning.
method Combining early stopping theory with attention-based Neural Tangent Kernel (NTK) for LLMs.
result Formalizes convergence rate of attention-based fine-tuning with respect to sample size.
Generalizes NTK for surrogate gradient learning in neural networks.
problem Lack of theoretical foundation for surrogate gradient learning.
method Generalizes neural tangent kernel (NTK) for surrogate gradient learning (SGL).
result Surrogate gradient NTK provides a good characterization of SGL.
The paper examines when NTK theory applies to real finite-width neural networks.
problem Understanding when NTK theory accurately predicts the behavior of finite-width neural networks.
method Empirical study of fully-connected ReLU and sigmoid DNNs with various hyperparameters and depths.
result NTK theory does not always apply to sufficiently deep networks with exploding gradients, and the kernel changes significantly during training.
The paper examines the neural tangent kernel for PINNs solving general PDEs and finds convergence conditions.
problem Analyzing the convergence of neural tangent kernel for PINNs solving general PDEs.
method Analysis of NTK initialization and convergence during training for general PDEs using PINNs.
result Homogeneity of differential operators is crucial for NTK convergence.
New method improves covariance estimation for weighted samples.
problem Improving covariance estimation for weighted sample data.
method Asymptotic non-linear shrinkage formulas for covariance and precision matrix estimators of weighted sample covariances.
result Asymptotic non-linear shrinkage formulas for covariance and precision matrix estimators of weighted sample covariances.
This paper explains robust overfitting in wide DNNs using adversarial training and NTK theory.
problem Robust overfitting in adversarially trained wide DNNs.
method Theoretical analysis using neural tangent kernel (NTK) theory and adversarial training dynamics.
result Adversarial training can lead to robust overfitting in wide DNNs, which can be mitigated by the proposed Adv-NTK method.
We study the concentration of NTK for MLPs at EOC, proving finite-width approximation of gradient independence.
problem Understanding the concentration of Neural Tangent Kernel (NTK) for MLPs at the Edge of Chaos (EOC).
method Proved approximate gradient independence holds at finite width, using maximal inequalities to show NTK matrix concentrates around its infinitely wide limit.
result The NTK matrix of MLPs at EOC concentrates around its infinitely wide limit, requiring hidden layer widths to grow quadratically.
This paper improves neural network learning by escaping the NTK regime and efficiently learning sparse polynomials.
problem Learning sparse polynomials efficiently using neural networks.
method Spectral analysis of NTK, identifying 'good' directions, and constructing a regularizer.
result Gradient descent on a two-layer neural network can learn sparse polynomials efficiently, improving over the NTK and QuadNTK.
Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to c…
The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectr…
Gradient descent learns over-param neural nets better than NTK.
problem Learning over-parametrized neural networks with ReLU activations.
method Gradient descent from random initialization on a Gaussian input distribution.
result Gradient descent achieves population loss , while NTK achieves .
Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelsstat.ML
New insights into model robustness for random features and NTK models.
problem Understanding and distinguishing robustness in machine learning models.
method Analyzing empirical risk minimization in random features and NTK models.
result Random features models are not robust under any degree of over-parameterization, even when satisfying the universal law of robustness.
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
problem Convergence analysis of averaged stochastic gradient descent for neural networks.
method Analyzed convergence of averaged stochastic gradient descent for overparameterized two-layer neural networks.
result Achieved minimax optimal convergence rate with global convergence guarantee.
Novel algorithms improve efficiency of finite width NTK computation.
problem Efficiency of computing finite width Neural Tangent Kernel (NTK).
method Leveraging neural network structure, propose two novel algorithms.
result Significantly improved efficiency in compute and memory requirements.
The problem of finding overlapping communities in networks has gained much attention recently. Optimization-based approaches use non-negative matrix factorization (NMF) or variants, but the global optimum cannot be provably attained in general. Model-based approaches, such as the popular mixed-membership stochastic blo…
This paper explains GCNs using NTKs and improves their performance.
problem GCNs' performance degrades with depth, and skip connections marginally improve it.
method Derive NTKs for GCNs, validate with simulations, propose NTK as a surrogate model.
result Suitable normalisation can prevent drastic performance drop with depth.
On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecs.LG
NTK theory fails to predict practical behavior of large-width neural networks.
problem Theoretical limits of NTK do not match practical neural network architectures.
method Empirical investigation of NTK's applicability to large-width architectures.
result Practically relevant behavior of large-width architectures differs from NTK theory.
A new method uses neural tangent kernel to efficiently compute MMD statistic.
problem Efficiently computing Maximum Mean Discrepancy (MMD) statistic with low memory and computational complexity.
method Identifies a connection between neural tangent kernel (NTK) and MMD to develop a computationally and memory-efficient approach.
result The proposed NTK-MMD statistic is validated through numerical experiments on synthetic and real-world datasets.
Sparse spectral decomposition identifies overlapping communities in networks.
problem Estimating overlapping community memberships in networks where nodes can belong to multiple communities.
method Sparse principal subspace estimation with iterative thresholding.
result The fixed point of the algorithm corresponds to correct node memberships under the stochastic block model.
New NTK bounds show deep networks with minimum over-parameterization can still memorize and optimize.
problem Understanding memorization and optimization in sub-linear over-parameterized deep networks.
method Lower bound on NTK eigenvalues for deep networks with minimum over-parameterization.
result Deep networks with minimum over-parameterization can still be powerful memorizers and optimizers.
New model for detecting communities in weighted bipartite networks.
problem No model for community detection in overlapping bipartite weighted networks.
method Introduces BiMMDF model allowing any distribution with block structure.
result Efficient algorithm with theoretical guarantee of consistent estimation.
Unified theory of ownership concentration, overlap, and dependence.
problem Understanding the complex layers of ownership concentration, overlap, and dependence in financial markets.
method Develops a unified quadratic framework for analyzing these layers and their interactions.
result Unified framework shows that the same residual operator measures static overlap and governs linearized market transmission.
This work introduces a novel estimation method, called LOVE, of the entries and structure of a loading matrix A in a sparse latent factor model X = AZ + E, for an observable random vector X in Rp, with correlated unobservable factors Z \in RK, with K unknown, and independent noise E. Each row of A is scaled and sparse.…
This paper analyzes convergence of FL for neural networks using NTK.
problem Theoretical guarantees of FL for neural networks with explicit forms and multi-step updates are unexplored.
method FL-NTK framework for federated learning of ReLU neural networks trained by gradient descent.
result FL-NTK converges to a global-optimal solution at a linear rate with proper learning parameters.