Neural networks outperform kernels by learning features better.
problem Current theories of feature learning do not adequately assess feature quality.
method Introduced feature quality metric and examined existing theories empirically.
result Current theories of feature learning do not provide a sufficient foundation for neural network generalization.
Paper identifies how neural networks learn features.
problem Understanding how neural networks automatically select features.
method Deep Neural Feature Ansatz, implementing average gradient outer product.
result Recursive Feature Machines achieve state-of-the-art performance.
Gradient descent learns useful features even in the NTK regime.
problem The ability of neural networks to learn useful features.
method Local convergence analysis of gradient descent with regularization.
result Gradient descent can capture ground-truth directions for feature learning even after the loss threshold is reached.
Deep neural networks are a powerful tool for feature learning and extraction given their ability to model high-level abstractions in highly complex data. One area worth exploring in feature learning and extraction using deep neural networks is efficient neural connectivity formation for faster feature learning and extr…
A new framework learns system design using neural features in function space.
problem Learning system design with neural feature extractors.
method Introduces feature geometry in function space, nesting technique for optimal feature approximation.
result Optimal features found from data samples using off-the-shelf architectures and optimizers.
Deep learning explained through spectral filtering of hierarchical features.
problem Understanding how deep neural networks learn useful representations from data.
method Neural Low-Degree Filtering (Neural LoFi) as a stylized limit of gradient-based training.
result Predicts how representations are selected layer by layer and explains emergence of concepts.
Derives FACT, an alternative to NFA for neural networks, explaining feature learning.
problem Understanding how neural networks learn representations.
method First-principles approach using first-order optimality conditions.
result FACT explains why NFA holds and provides a principled alternative.
Graph neural network (GNN) is a deep model for graph representation learning. One advantage of graph neural network is its ability to incorporate node features into the learning process. However, this prevents graph neural network from being applied into featureless graphs. In this paper, we first analyze the effects o…
Gradient descent with growing learning rate enables learning non-linear features in neural networks.
problem Learning non-linear features in two-layer neural networks.
method Using gradient descent with a learning rate that grows with the sample size.
result Multiple rank-one components emerge, each corresponding to a specific polynomial feature.
Anisotropic neural network selects relevant features from datasets.
problem Reduction of irrelevant features improves model interpretability and performance.
method General Regression Neural Network with an anisotropic Gaussian Kernel.
result The method robustly selects features from simulated and real-world datasets.
This paper shows how deep neural networks can learn rich, independent features that significantly deviate from initialization.
problem Understanding how deep neural networks achieve meaningful feature learning and global convergence.
method Investigation of infinitely wide, L-layer neural networks using the tensor program framework under Maximal Update parametrization. result SGD enables these networks to learn linearly independent features that substantially deviate from their initial values, capturing relevant data information.
Training shapes the geometry of neural network feature maps, revealing local area magnification.
problem Understanding how training affects the geometric structure of neural network feature maps.
method Analyzing the Riemannian geometry induced by neural network feature maps at infinite width and after training.
result Training breaks the symmetry of the geometry induced by random neural network feature maps, magnifying local areas along decision boundaries.
Gradient descent aligns neural feature matrices with pre-activation tangent features.
problem Understanding neural feature learning mechanisms.
method Analytical proof of alignment between weight matrices and pre-activation tangent features.
result Derivative alignment occurs almost surely in high-dimensional settings.
Wide neural networks with asymmetrical node scaling converge globally and learn features.
problem Global convergence and feature learning in over-parameterised shallow networks.
method Gradient-based optimisation of wide, shallow neural networks with asymmetrical node scaling.
result Gradient flow and gradient descent converge to a global minimum and learn features, unlike in the NTK parameterisation.
This work studies how contrastive learning extracts features from unlabeled data.
problem How neural networks trained by contrastive learning can extract features from unlabeled data.
method Formal analysis of contrastive learning's feature learning process, considering two types of features: sparse and dense.
result Contrastive learning using ReLU networks can learn sparse features if proper augmentations are adopted.
Recursive Feature Machines show grokking in modular arithmetic without neural networks.
problem Grokking in modular arithmetic tasks.
method Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP).
result RFM and neural networks learn block-circulant features to solve modular arithmetic.
In recent years, it has been found that neural networks can be easily fooled by adversarial examples, which is a potential safety hazard in some safety-critical applications. Many researchers have proposed various method to make neural networks more robust to white-box adversarial attacks, but an effective method have …
Graph neural network using Beltrami flow for feature and topology evolution.
problem Efficient feature learning and topology evolution on graphs.
method Discretized Beltrami flow applied to graph neural networks with positional encodings.
result Achieves state-of-the-art results on various benchmarks.
Derives a biologically plausible neural network for Slow Feature Analysis.
problem Learning latent features from time series data.
method Starting from an SFA objective, derives Bio-SFA with a biologically plausible neural network implementation.
result Validates Bio-SFA on naturalistic stimuli, reproducing interesting properties of brain cells.
This paper presents an unsupervised method to learn a neural network, namely an explainer, to interpret a pre-trained convolutional neural network (CNN), i.e., the explainer uses interpretable visual concepts to explain features in middle conv-layers of a CNN. Given feature maps of a conv-layer of the CNN, the explaine…
Inspired by the immense success of deep learning, graph neural networks (GNNs) are widely used to learn powerful node representations and have demonstrated promising performance on different graph learning tasks. However, most real-world graphs often come with high-dimensional and sparse node features, rendering the le…
Neural networks in many varieties are touted as very powerful machine learning tools because of their ability to distill large amounts of information from different forms of data, extracting complex features and enabling powerful classification abilities. In this study, we use neural networks to extract features from b…
Model shows feature learning can improve neural scaling laws for hard tasks.
problem Understanding and improving neural network scaling laws for various task difficulties.
method Developed a solvable model of neural scaling laws, identified three scaling regimes, and demonstrated feature learning's impact on scaling exponents.
result Feature learning can improve scaling with training time and compute for hard tasks, nearly doubling the exponent.
Neural networks learn task-specific features, influenced by nonlinearity.
problem Understanding the nature of task-dependent feature learning in neural networks.
method Investigation of fully-connected, wide neural networks using Bayesian framework.
result The nature of internal representations depends on neuronal nonlinearity, leading to analog, redundant, or sparse coding schemes.
Recently, over-parameterized neural networks have been extensively analyzed in the literature. However, the previous studies cannot satisfactorily explain why fully trained neural networks are successful in practice. In this paper, we present a new theoretical framework for analyzing over-parameterized neural networks …
Study reveals conditions for neural networks to forget learned features.
problem Understanding feature unlearning in neural networks.
method Infinite-width limit analysis with stochastic gradient descent, fast-slow dynamics.
result Conditions for feature unlearning are determined by the strength of nonlinear terms and initial weights.
Three-layer networks learn more complex features than two-layer networks.
problem Understanding feature learning in deep neural networks.
method Analysis of three-layer neural networks trained with gradient descent.
result Three-layer networks can learn functions that two-layer networks cannot.
Adaptive Group Lasso selects important features in neural networks.
problem Lack of interpretability in neural networks.
method Adaptive Group Lasso for feature selection.
result Consistent feature selection for neural networks with theoretical guarantee.
dGAP learns feature dependencies and predicts targets simultaneously.
problem Learning task-agnostic statistical dependencies and missing explicit feature dependencies.
method Jointly optimizes a neural dependency graph and target prediction loss.
result dGAP can recover correct feature dependencies and improve prediction accuracy.
Study links neural network inductive bias, feature learning, and generalization on Boolean functions.
problem Understanding how neural networks learn and generalize on Boolean data.
method End-to-end analysis of depth-2 discrete fully connected networks and DNF formulas, using Monte Carlo learning.
result Predictable training dynamics and interpretable features emerge, linking inductive bias and generalization.
NGMs create mirrored features to assess neural network feature importance.
problem Lack of feature relevance information in DNNs limits their applicability.
method Structured perturbation and kernel-based conditional dependence measure for feature importance evaluation.
result Controls feature selection error rate and maintains high selection power with correlated features.
Neural collapse occurs in normalized features over a Riemannian manifold.
problem Understanding neural collapse in normalized feature models.
method Simplified multi-class classification task to a nonconvex optimization problem over the Riemannian manifold, analyzing the landscape of critical points.
result The only global minimizers are neural collapse solutions, with all other critical points being strict saddles.
Regularization effect found in neural feature alignment.
problem Implicit regularization in deep learning models.
method Geometrical viewpoint and analysis of Rademacher complexity.
result Neural features align along task-relevant directions, leading to regularization.
Feature networks link ML features via graph structure for enhanced learning.
problem Enhancing feature expressiveness and learning efficiency in machine learning.
method Graph representation of feature vectors, leveraging Fourier and functional analysis.
result Feature networks enable novel, complex feature dependencies.
AGF explains feature learning in neural networks through alternating steps.
problem Understanding what features neural networks learn and how they learn them.
method AGF is an algorithmic framework that approximates the dynamics of feature learning in two-layer networks.
result AGF provides a unified framework to understand feature learning in neural networks, matching experimental results across various architectures.
AGOP mechanism explains deep neural collapse in neural networks.
problem Explaining the rigid structure of data representations in deep neural networks.
method Introducing AGOP and Deep RFM to demonstrate DNC.
result AGOP mechanism causes deep neural collapse in neural networks.
Artificial Neural Networks form the basis of very powerful learning methods. It has been observed that a naive application of fully connected neural networks to data with many irrelevant variables often leads to overfitting. In an attempt to circumvent this issue, a prior knowledge pertaining to what features are relev…
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
Paper proposes FINN for better CTR prediction by modeling feature interactions.
problem Improving CTR prediction in advertising and recommendation systems.
method Proposes Feature Interaction based Neural Network (FINN) to model feature interactions.
result FINN outperforms other deep models in CTR prediction tasks.
Random matrix theory explains how neural networks adapt to data.
problem Understanding how neural networks learn and generalize from data.
method Random matrix analysis of two-layer neural networks.
result Sharp characterization of feature spectrum and generalization error.
Supporting evidence for adaptive feature program across diverse models.
problem Analyzing feature learning in neural networks.
method Over-parameterized sequence models and feature error measure (FEM).
result FEM is decreasing during training of adaptive feature models.
Gradient Starvation causes neural networks to focus on a subset of features, leading to imbalance.
problem Neural networks focusing on only a subset of features despite having access to more.
method Using Dynamical Systems theory, identified properties of learning dynamics leading to feature imbalance.
result Gradient descent can lead to feature imbalance in over-parameterized networks.
Although the recent progress in the deep neural network has led to the development of learnable local feature descriptors, there is no explicit answer for estimation of the necessary size of a neural network. Specifically, the local feature is represented in a low dimensional space, so the neural network should have mo…
New mechanism discovered for feature learning in CNNs.
problem Understanding how CNNs learn features from images.
method Proposed Convolutional Neural Feature Ansatz linking filter covariances to patch-based AGOPs.
result Deep ConvRFM algorithm learns features similar to deep CNNs, improving performance.
Random neural nets learn Black-Scholes PDEs without dimensionality issues.
problem Learning Black-Scholes type PDEs efficiently in high dimensions.
method Random feature neural networks applied to Kolmogorov PDEs.
result Random neural nets avoid the curse of dimensionality for Black-Scholes PDEs.
Wide neural networks converge linearly to zero loss with feature learning.
problem Optimizing wide neural networks with feature learning guarantees.
method Gradient flow analysis for wide shallow and multi-layer NNs.
result Training loss converges linearly to zero for wide NNs under GF, demonstrating feature learning and better generalization.
Temporal-difference and Q-learning learn feature representations that converge to optimal ones.
problem Understanding how feature representations evolve in temporal-difference and Q-learning with neural networks.
method Mean-field theory applied to overparameterized two-layer neural networks.
result The feature representation converges to the optimal one, generalizing previous results.
Data augmentation affects feature importance, enhancing learning for neural networks.
problem Understanding the effect of data augmentation on feature importance and learning dynamics.
method Analyzing a two-layer convolutional neural network in a multi-view data model.
result Data augmentation alters feature importance, making certain features more likely to be learned.