Neural networks benefit from intermediate representations, reducing sample complexity.
problem Understanding how neural networks leverage intermediate representations for hierarchical learning.
method Fixed, randomly initialized neural network as a representation function, compared with raw inputs and other trainable networks.
result Neural representations can achieve improved sample complexities compared to raw inputs, especially for low-rank polynomials.
Enhanced BCPNN learns hidden representations without labels.
problem Unsupervised learning of hierarchical representations in neural networks.
method Extended BCPNN architecture with local Hebbian learning mechanisms.
result Demonstrated capability for unsupervised learning of salient hidden representations.
GNNs learn graph representations, with new theory on their power and limitations.
problem Understanding the capabilities and limitations of GNNs.
method Theoretical analysis of GNNs, focusing on approximation and learning properties.
result New insights into the representation, generalization, and extrapolation of GNNs.
A neural scene representation framework enforcing 3D transformations.
problem Learning 3D scene representations from images without 3D supervision.
method Introducing a loss enforcing equivariance of the scene representation with 3D transformations.
result Real-time neural rendering with comparable results to models requiring minutes for inference.
Neural nets learn robust geometric data representations.
problem Ensuring neural networks are robust to adversarial attacks.
method Topological Data Analysis via persistence diagrams, Lipschitz stability.
result Certified ε-robustness on ORBIT5K dataset. Neural networks need guidance to learn appropriate internal representations.
problem Neural networks struggle to learn appropriate internal representations without guidance.
method Encouraging an appropriate internal representation improves neural network performance.
result Guiding neural networks to learn appropriate internal representations improves their performance.
We study how finite Bayesian neural networks adapt their hidden representations.
problem Understanding how finite Bayesian neural networks differ from infinite ones.
method We analyze the asymptotics of learned feature kernels for various network architectures.
result The leading finite-width corrections to feature kernels have a universal form.
New approach uses prior knowledge to improve neural network representations.
problem Difficulty in learning robust internal representations without a large training set.
method Incorporates prior information into a pre-trained reasoning module.
result Improves representation learning in diverse self-supervised learning settings.
Deconfounds neural network representation similarity metrics to improve consistency and accuracy.
problem Confounding by population structure in similarity metrics like RSA and CKA.
method Covariate adjustment regression to adjust for confounders.
result Improves detection of semantically similar neural networks and consistency in transfer learning.
The study analyzes neural network predictions of knot invariants and finds that braid representations work best.
problem Understanding and predicting knot invariants using neural networks.
method Investigated different knot representations and invariants, proposed a cosine similarity score.
result Braid representations are best for predicting knot invariants, and some invariants are easier to learn than others.
Neural networks learn distance-based representations, not just intensity.
problem Understanding how neural networks interpret and learn from internal activations.
method Manipulated ReLU and Absolute Value activations to observe sensitivity to distance and intensity perturbations.
result Neural networks are highly sensitive to small distance-based perturbations, challenging the intensity-based interpretation.
Neural networks learn spectral representations for group composition.
problem Understanding structured emergence in neural network training.
method Lifting gradient flow to Fourier domain, proving convergence to irreducible representations.
result Neurons converge to single irreducible representations, cross-layer coefficients align.
A new framework for fair representation learning using correction vectors.
problem Learning fair representations that are independent of sensitive features.
method Introducing correction vectors to neural network features for fair representation learning.
result The approach does not impact performance while ensuring fairness.
This work analyzes how neural networks learn representations in actor-critic algorithms.
problem Theoretical support for neural AC algorithms is limited to linear function approximations.
method Mean-field analysis of a two-timescale learning AC algorithm with overparameterized networks.
result Neural AC finds the globally optimal policy at a sublinear rate in the continuous-time and infinite-width limiting regime.
MetaFun learns functional representations for meta-learning.
problem Few-shot classification on large-scale datasets.
method Functional encoder-decoder approach with iterative updates.
result State-of-the-art performance on miniImageNet and tieredImageNet.
DORA analyzes deep neural networks' internal representations to detect spurious correlations.
problem Detecting spurious correlations in deep neural networks' internal representations.
method DORA uses Extreme-Activation (EA) distance measure to assess representation similarities.
result Identifies internal representations capable of detecting spurious correlations.
Introduces TT-NF for more compact neural field representations.
problem Finding more compact and easy-to-fit neural field representations.
method Tensor Train parameterization trained with backpropagation.
result Low-rank compression improves downstream task quality metrics.
Temporal-difference and Q-learning learn feature representations that converge to optimal ones.
problem Understanding how feature representations evolve in temporal-difference and Q-learning with neural networks.
method Mean-field theory applied to overparameterized two-layer neural networks.
result The feature representation converges to the optimal one, generalizing previous results.
Neural networks can learn distance metrics affecting model performance.
problem Understanding how neural networks learn and represent data.
method Experiments with six MNIST architectures, constrained to learn either distance or intensity representations.
result Distance-based learning affects model performance, validating the geometric framework.
Neural networks learn task-specific features, influenced by nonlinearity.
problem Understanding the nature of task-dependent feature learning in neural networks.
method Investigation of fully-connected, wide neural networks using Bayesian framework.
result The nature of internal representations depends on neuronal nonlinearity, leading to analog, redundant, or sparse coding schemes.
Meta-learning neural networks for better clustering representations.
problem Improving clustering performance with appropriate representations.
method Meta-learning method that trains neural networks for representations using VB inference with an infinite Gaussian mixture model.
result The method achieves higher clustering performance than existing methods.
Graph convolutional deep kernel machine learns representations for graph tasks.
problem Limited representation learning in infinite-width neural networks.
method Developed a graph convolutional deep kernel machine as an infinite-width limit.
result Representation learning improves performance for heterophilous node classification tasks.
Neural networks can learn useful representations that kernels can't.
problem Learning functions that depend on only a few relevant directions.
method Gradient descent on a two-layer neural network.
result Improved sample complexity for learning polynomials.
Math theory explains how neural networks learn abstract representations.
problem Understanding how neural networks learn abstract representations.
method Mathematical theory reformulating network optimization into mean field optimization over neural preactivations.
result Abstract representations of latent variables are guaranteed to appear in neural networks trained on tasks that depend on these variables.
GGAN improves audio representation learning with fewer labels.
problem Learning representations for specific tasks from unlabelled data.
method Guided Generative Adversarial Neural Network (GGAN).
result GGAN learns better representations with fewer labelled data.
INVERT connects neural representations to human-understandable concepts.
problem Lack of understanding and statistical significance in existing explainability methods.
method Inverse Recognition (INVERT) approach that connects learned representations to human-understandable concepts.
result INVERT provides interpretable metrics and statistical significance for representation alignment.
New method learns representations for decision forests using input perturbation.
problem Decision forests struggle with raw structured data and lack effective representations.
method Approximate decision forest gradients through input perturbation.
result Effective representation learning for decision forests without structural changes.
Defines metrics to compare neural network representations.
problem Comparing neural network representations across different architectures and tasks.
method Developed a family of metric spaces and modified existing measures to quantify representational dissimilarity.
result Identified relationships between neural representations and anatomical features.
Geo2DR learns graph representations using substructure patterns.
problem Learning distributed representations of graphs efficiently.
method Unsupervised learning with discrete substructure patterns and neural language models.
result Geo2DR achieves high reproducibility and interoperability in graph classification.
Unified taxonomy for graph representation learning.
problem Lack of unified understanding and integration of graph representation learning methods.
method Proposes a Graph Encoder Decoder Model (GRAPHEDM) to unify graph neural networks, network embedding, and graph regularization.
result Unified taxonomy and Graph Encoder Decoder Model (GRAPHEDM) for graph representation learning.
Deep Learning has been shown to learn efficient representations for structured data such as image, text or audio. In this chapter, we present neural network architectures that are able to learn efficient representations of molecules and materials. In particular, the continuous-filter convolutional network SchNet accura…
GraphCL learns node representations by maximizing similarity between perturbed node features.
problem Learning node representations in graph data without labeled data.
method Contrastive learning of node embeddings using graph neural networks and a loss function.
result Significantly outperforms state-of-the-art in unsupervised node classification benchmarks.
This work characterizes how data augmentation shapes neural representations.
problem Understanding the impact of data augmentation on neural network representations.
method Embedding neural network hidden representations into a metric space invariant to transformations, analyzing shape-space trajectories.
result Increasing data augmentation strength leads to well-behaved trajectories in the embedded space, and different augmentation types steer representations in distinct directions.
DIFNET tackles the suspended animation problem in deep graph neural networks.
problem Deep graph neural networks suffer from the suspended animation problem.
method DIFNET uses neural gates and graph residual learning for node hidden state modeling, and includes an attention mechanism for node neighborhood information diffusion.
result DIFNET effectively addresses the suspended animation problem and improves learning performance.
Inspired by the immense success of deep learning, graph neural networks (GNNs) are widely used to learn powerful node representations and have demonstrated promising performance on different graph learning tasks. However, most real-world graphs often come with high-dimensional and sparse node features, rendering the le…
RedEx improves neural network optimization with convex optimization guarantees.
problem Difficult optimization of neural networks.
method RedEx architecture using convex optimization with semi-definite constraints.
result RedEx can efficiently learn functions fixed methods cannot.
MSA compares neural representations' intrinsic geometry for better understanding.
problem Existing similarity measures fail to capture subtle distinctions between neural network solutions.
method Metric similarity analysis (MSA) using Riemannian geometry.
result MSA can disentangle features of neural computations and compare nonlinear dynamics.
Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to solve this issue by pr…
HGP-SL pools and learns graph structure for hierarchical representation learning.
problem Graph pooling is overlooked in GNN models, limiting hierarchical representation learning.
method Integrates graph pooling and structure learning into a unified module.
result HGP-SL improves graph classification performance on benchmarks.
Deep neural network learns compact representations for driving tasks.
problem Improving autonomous driving through better neural network representations.
method Inspired by human brain's hierarchical structure and predictive nature, the paper proposes a deep learning framework that learns compact representations of driving concepts.
result The paper successfully learns compact representations using as few as 16 neural units for car and lane concepts.
Deep learning explained through spectral filtering of hierarchical features.
problem Understanding how deep neural networks learn useful representations from data.
method Neural Low-Degree Filtering (Neural LoFi) as a stylized limit of gradient-based training.
result Predicts how representations are selected layer by layer and explains emergence of concepts.
Disentangled representations naturally emerge in multi-task learning.
problem Finding adaptable representations for multiple tasks.
method Empirical study of neural networks trained on automatically generated supervised tasks.
result Disentanglement naturally occurs during multi-task learning.
A novel feature representation method for non-image based features.
problem Inability of Convolutional Neural Networks for non-image based features or features without spatial correlations.
method REFINED: Representation of Features as Images with Neighborhood Dependencies.
result Higher prediction accuracy compared to existing methodologies.
Unsupervised representation learning has succeeded with excellent results in many applications. It is an especially powerful tool to learn a good representation of environments with partial or noisy observations. In partially observable domains it is important for the representation to encode a belief state, a sufficie…
We consider deep feedforward neural networks with rectified linear units from a signal processing perspective. In this view, such representations mark the transition from using a single (data-driven) linear representation to utilizing a large collection of affine linear representations tailored to particular regions of…
Deep Neural Networks have shown tremendous success in the area of object recognition, image classification and natural language processing. However, designing optimal Neural Network architectures that can learn and output arbitrary graphs is an ongoing research problem. The objective of this survey is to summarize and …
New algorithm learns invariant representations for robust neural networks.
problem Learning robust neural network representations that are invariant to certain factors.
method Causal perspective and distribution matching approach.
result Empirically, the algorithm achieves state-of-the-art performance on domain generalization.
DRIFT uses neural flows to replace distributional regression models.
problem Lack of neural network representations for distributional regression models.
method Inverse flow transformations (DRIFT) for distributional regression.
result Neural representations in DRIFT match classical statistical methods in performance.