Unified understanding of neural representation similarity measures.
problem Fragmented research landscape of neural network similarity measures.
method Observation and exploration of connections between shape distances and normalized Bures similarity.
result Cosine of the Riemannian shape distance equals normalized Bures similarity.
Deconfounds neural network representation similarity metrics to improve consistency and accuracy.
problem Confounding by population structure in similarity metrics like RSA and CKA.
method Covariate adjustment regression to adjust for confounders.
result Improves detection of semantically similar neural networks and consistency in transfer learning.
Geometric stability measures neural network robustness, distinguishing from similarity metrics.
problem Lack of robustness in neural network representations.
method Introduces geometric stability, quantified by Shesha metric measuring self-consistency.
result Stability and similarity are uncorrelated, revealing distinct properties of neural network robustness.
Neural networks are capable of learning rich, nonlinear feature representations shown to be beneficial in many predictive tasks. In this work, we use such models to explore different geographical feature representations in the context of predicting colorectal cancer survival curves for patients in the state of Iowa, sp…
Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for comparing neural network representations based on canonical correlation analysis (CCA). We show that CCA belongs to a family of statistics for mea…
The paper analyzes MAML's representation using RSA, revealing that feature reuse is not the primary reason for its success.
problem Understanding why model-agnostic meta-learning (MAML) works well in few-shot learning tasks.
method Representation similarity analysis (RSA) applied to MAML's few-shot learning instantiation.
result Feature reuse is not the primary reason for MAML's success; instead, it is the learning task itself that increases representation similarity.
Novel tRSA combines geometry and topology for brain and model analysis.
problem Traditional RSA overlooks topological information in neural representations.
method Topological RSA (tRSA) using nonlinear monotonic transforms.
result Robust model comparisons and novel insights into neural computation.
Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distribution…
New measures link neural representation geometry to decoding ability.
problem Understanding how neural representations relate to decoding ability.
method Showed that popular similarity measures can be interpreted from a decoding perspective.
result Proved that measures like CKA and CCA quantify alignment between optimal linear readouts.
New method learns behavioral representations from mobility data.
problem Analyzing behavioral similarity of moving individuals from CDR trajectories.
method mob2vec framework combining segmentation, generalization, and unsupervised learning.
result Mob2vec generates low-dimensional vector representations preserving mobility behavior similarities.
Unified toolkit for comparing neural representations using SRTD and NTS.
problem Heuristic asymmetry and unbounded scores in existing divergences.
method Developed SRTD and NTS to address these issues.
result Unified, robust, and scale-invariant metric for comparing neural representations.
Modified RV-coefficient reveals how training affects neural network representations.
problem Understanding how training affects intermediate representations in convolutional neural networks.
method Experimented with modified RV-coefficient (RV2) to compare activation patterns in deep networks trained on varying amounts of data and layers.
result RV2 successfully recovered expected similarity patterns and provided interpretable similarity matrices.
A new method identifies similar mutual funds using graph learning.
problem Identifying similar mutual funds with nuanced portfolio similarities.
method Node2Vec machine learning method applied to a weighted bipartite network of funds and assets.
result Identifies structural similarity among mutual funds' portfolios.
MSA compares neural representations' intrinsic geometry for better understanding.
problem Existing similarity measures fail to capture subtle distinctions between neural network solutions.
method Metric similarity analysis (MSA) using Riemannian geometry.
result MSA can disentangle features of neural computations and compare nonlinear dynamics.
The paper studies how neural networks evolve representations, finding a unique fixed point for nonlinear activations.
problem Understanding how neural networks transform input data across layers.
method Theoretical framework for the evolution of the kernel sequence, using mean-field regime and Hermite polynomials.
result For nonlinear activations, the kernel sequence converges globally to a unique fixed point.
New framework to test neural network representation similarity measures.
problem Disagreements among dissimilarity measures in neural networks.
method Statistical testing framework to evaluate measures based on functional behavior.
result Current metrics have different weaknesses; a classical baseline performs surprisingly well.
New findings on representation changes in transfer learning.
problem Understanding representation changes in transfer learning.
method Analysis of pre-trained and fine-tuned representations in transfer learning.
result Representation changes mainly occur in the head, but early layers are also important for cross-domain adaptation.
Correlated component analysis as proposed by Dmochowski et al. (2012) is a tool for investigating brain process similarity in the responses to multiple views of a given stimulus. Correlated components are identified under the assumption that the involved spatial networks are identical. Here we propose a hierarchical pr…
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
PRESTO maps latent representations across diverse ML models.
problem Understanding variability in latent representations across different ML models.
method Uses persistent homology to characterize latent spaces and measure their pairwise similarity.
result Preserves desirable properties and enables sensitivity analysis of latent representations.
Defines metrics to compare neural network representations.
problem Comparing neural network representations across different architectures and tasks.
method Developed a family of metric spaces and modified existing measures to quantify representational dissimilarity.
result Identified relationships between neural representations and anatomical features.
Representations of C∗-algebras are realized on section spaces of holomorphic homogeneous vector bundles. The corresponding section spaces are investigated by means of a new notion of reproducing kernel, suitable for dealing with involutive diffeomorphisms defined on the base spaces of the bundles. Applications of th…
Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "n…
SAFE detects fake news by analyzing text and image similarities.
problem Detecting fake news with less focus on text-image similarity.
method SAFE uses neural networks to extract text and visual features, then learns their relationship to predict fake news.
result SAFE effectively recognizes fake news based on text, images, or mismatches.
Proposes a method to compare neural networks using feature and gradient vectors.
problem Understanding the behavior of neural networks trained on different datasets.
method Defines a similarity index using feature and gradient vectors, and employs sketching techniques for efficient comparison.
result Demonstrates superior performance in computing similarity of neural networks trained on different datasets.
Comparing different neural network representations and determining how representations evolve over time remain challenging open questions in our understanding of the function of neural networks. Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, e…
New findings clarify the link between distributional closeness and representational similarity.
problem When and why do different neural network representations become similar?
method Identifiability theory, focusing on model families including autoregressive language models.
result Small Kullback-Leibler divergence does not guarantee similar representations.
Paper analyzes how contrastive learning structures learned representations.
problem Understanding the structure of learned representations in contrastive learning.
method Kernel-based contrastive learning framework (KCL) and statistical dependency viewpoint.
result Theoretical upper bound and generalization error bound for KCL.
Simplifies fair PCA with fast, efficient solution.
problem Learning fair low-rank approximations of data.
method Conceptually simple approach with analytic solution.
result Faster and similar results to existing fair PCA methods.
Graph neural networks improve with edge similarity constraints in RNA structure analysis.
problem Lack of edge similarity constraints in graph neural networks.
method Introduced a graph neural network layer that leverages prior information about edge similarities.
result Edge similarity constraints do not enhance performance in graph neural networks.
The autoencoder is an artificial neural network model that learns hidden representations of unlabeled data. With a linear transfer function it is similar to the principal component analysis (PCA). While both methods use weight vectors for linear transformations, the autoencoder does not come with any indication similar…
New method learns low-dimensional representations of AI-generated treatments.
problem Representing AI-generated treatments without losing semantic meaning.
method Double kernel representation learning with alternating minimization.
result Efficiently learned representations guide generative models and facilitate adaptive online experiments.
Representational Similarity Analysis (RSA) aims to explore similarities between neural activities of different stimuli. Classical RSA techniques employ the inverse of the covariance matrix to explore a linear model between the neural activities and task events. However, calculating the inverse of a large-scale covarian…
Develops an ordinal-similarity framework for scalable and interpretable representation alignment.
problem Measuring representation similarity in large datasets.
method Triplet and Quadruplet Similarity Indices.
result Demonstrates inherent interpretability, robustness to outliers, and computational efficiency.
One of the ubiquitous representation of long DNA sequence is dividing it into shorter k-mer components. Unfortunately, the straightforward vector encoding of k-mer as a one-hot vector is vulnerable to the curse of dimensionality. Worse yet, the distance between any pair of one-hot vectors is equidistant. This is partic…
Feature normalization prevents collapse in non-contrastive learning dynamics.
problem Non-contrastive learning can collapse into a single point due to lack of repulsive force.
method Extended previous theory based on L2 loss to cosine loss, considering feature normalization.
result Cosine loss induces stable equilibrium, preventing collapse even with insufficient repulsive force.
AI models aligned with human vision perform well on few data tasks.
problem Few-shot learning performance with limited data.
method Information-theoretic analysis and empirical testing of 491 models.
result Highly aligned models show better robustness to attacks and domain shifts.
The standard loss function used to train neural network classifiers, categorical cross-entropy (CCE), seeks to maximize accuracy on the training data; building useful representations is not a necessary byproduct of this objective. In this work, we propose clustering-oriented representation learning (COREL) as an altern…
Learning a similarity metric has gained much attention recently, where the goal is to learn a function that maps input patterns to a target space while preserving the semantic distance in the input space. While most related work focused on images, we focus instead on learning a similarity metric for neuroimages, such a…
Deep Belief Network predicts lncRNA-disease associations with high accuracy.
problem Accurately identifying lncRNA-disease associations to understand lncRNA functionality and disease mechanism.
method Proposes a DBN-based model using heterogeneous networks and DBN for feature learning.
result Obtained AUC of 0.96 and AUPR of 0.967 on standard dataset.
DORA analyzes deep neural networks' internal representations to detect spurious correlations.
problem Detecting spurious correlations in deep neural networks' internal representations.
method DORA uses Extreme-Activation (EA) distance measure to assess representation similarities.
result Identifies internal representations capable of detecting spurious correlations.
A novel unsupervised feature selection method using subspace clustering and self-expressive model.
problem Feature selection for large datasets with minimal labeling effort.
method Subspace clustering with adaptive representation learning and regularized regression.
result The method effectively captures sample similarities and discriminative information.
New metric for disentangling multivariate representations, accounting for more complex entanglements.
problem Current disentanglement metrics fail to detect entanglements involving more than two variables.
method Partial Information Decomposition framework to analyze information sharing and propose a new disentanglement metric.
result The proposed metric correctly identifies entanglements in high-dimensional spaces.
Diffusion Maps framework is a kernel based method for manifold learning and data analysis that defines diffusion similarities by imposing a Markovian process on the given dataset. Analysis by this process uncovers the intrinsic geometric structures in the data. Recently, it was suggested to replace the standard kernel …
Representational similarity analysis (RSA) has been shown to be an effective framework to characterize brain-activity profiles and deep neural network activations as representational geometry by computing the pairwise distances of the response patterns as a representational dissimilarity matrix (RDM). However, how to p…
Extract common latent factors from graphs for better representation learning.
problem Graph-level representation learning challenges due to limited labeled data and poor negative sample selection.
method Graph-wise Common Latent Factor Extraction (GCFX) using deepGCFX model.
result Improved graph-level and node-level tasks performance compared to state-of-the-art methods.
We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings. We show an efficient learning algorithm based on stochastic proximal methods that is significantly faster than previous approaches, making it possible to perfor…
Logit distance bounds representational similarity of models.
problem Approximating linear similarity when distributions are close.
method Defined a logit distance and proved its relationship to representational dissimilarity.
result Logit distance bounds representational similarity, providing nontrivial control in practice.