This work explores how feature decorrelation improves self-supervised learning.
problem Complete and dimensional collapse issues in self-supervised learning.
method Study of a concise framework and connection to feature decorrelation.
result Feature decorrelation improves self-supervised learning.
Contrastive learning struggles with class collapse and feature suppression, revealing bias towards simpler solutions.
problem Contrastive learning struggles with class collapse and feature suppression, especially in supervised and unsupervised settings.
method Unified theoretical framework to determine which features are learnt by CL, revealing bias towards simpler solutions.
result Bias towards simpler solutions is a key factor in class collapse and feature suppression.
This paper extends neural collapse to regression problems, revealing key features and structures.
problem Understanding the structure learned by deep neural networks in regression tasks.
method Established Neural Regression Collapse (NRC) across different models, analyzing feature and weight alignments.
result Deep neural regression models exhibit a collapsed feature space, aligning with target dimensions and covariances.
Neural collapse occurs in normalized features over a Riemannian manifold.
problem Understanding neural collapse in normalized feature models.
method Simplified multi-class classification task to a nonconvex optimization problem over the Riemannian manifold, analyzing the landscape of critical points.
result The only global minimizers are neural collapse solutions, with all other critical points being strict saddles.
This paper extends neural collapse to imbalanced data under cross-entropy loss.
problem Analyzing neural collapse in deep networks with imbalanced data.
method Using the unconstrained feature model and cross-entropy loss, the paper studies neural collapse in imbalanced datasets.
result Feature vectors within the same class collapse to a single mean vector, but angles between them depend on sample size.
Study on Neural Collapse limits in deep learning.
problem Understanding the limits of Neural Collapse in deep learning.
method Investigated Neural Collapse in the context of generalization and feature learning, refining conjectures and conducting experiments.
result Neural Collapse primarily occurs on the train set and not on the test set, suggesting it is an optimization phenomenon with unclear connections to generalization.
Feature normalization prevents collapse in non-contrastive learning dynamics.
problem Non-contrastive learning can collapse into a single point due to lack of repulsive force.
method Extended previous theory based on L2 loss to cosine loss, considering feature normalization.
result Cosine loss induces stable equilibrium, preventing collapse even with insufficient repulsive force.
New findings suggest non-contrastive learning has many bad minima, not just collapsed ones.
problem The effectiveness of non-contrastive learning in unsupervised feature learning.
method Theoretical analysis and controlled experiments on simple data models.
result Non-contrastive losses have a preponderance of non-collapsed bad minima, and these minima are not avoided during training.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
Wide neural networks with weight decay exhibit neural collapse.
problem Proving neural collapse in wide neural networks trained with weight decay.
method Generic guarantees on neural collapse for wide networks with weight decay, proving low training error and balancedness, and bounded conditioning.
result First proof of neural collapse in end-to-end training of wide neural networks with weight decay.
Deep linear networks exhibit collapsing features and classifiers across datasets.
problem Understanding the collapse of features and classifiers in deep linear networks.
method Theoretical and empirical analysis of deep linear networks with MSE and CE losses.
result Deep linear networks exhibit NC properties, collapsing features and classifiers to orthogonal vectors.
We analyze neural collapse in neural networks, showing that features collapse to vertices of a Simplex ETF.
problem Understanding and optimizing the features learned in the last layer of neural networks during training.
method Simplified unconstrained feature model, studying the global optimization landscape of cross-entropy loss with weight decay.
result The global minimizers of the loss are Simplex ETFs, and other critical points are strict saddles with negative curvature.
This paper extends neural collapse to class-imbalanced datasets using an unconstrained ReLU feature model.
problem Understanding neural collapse in class-imbalanced datasets with cross-entropy loss.
method Generalized neural collapse to class-imbalanced settings using an unconstrained ReLU feature model.
result Class-means converge to orthogonal vectors with different lengths, and classifier weights align to these vectors.
Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.
problem Understanding Neural Collapse in MSE-trained deep nets.
method Developed a new MSE loss decomposition and introduced the central path concept.
result Exact dynamics of Neural Collapse along the central path can be predicted.
Our paper explains deep neural collapse in multiple layers.
problem Understanding deep neural collapse in multi-layered neural networks.
method Generalized unconstrained features model for deep networks.
result Deep unconstrained features model exhibits deep neural collapse.
Study shows 'Ordinal Neural Collapse' in deep OR tasks, revealing simple geometric relationships.
problem Understanding neural collapse in deep Ordinal Regression tasks.
method Combining cumulative link models and Unconstrained Feature Model to investigate neural collapse.
result Demonstrates 'Ordinal Neural Collapse' (ONC) with three key properties.
This paper examines how different loss functions affect neural network features and performance.
problem Investigating which loss function is best for deep neural networks.
method Examining last-layer features of deep networks and drawing inspiration from the Neural Collapse phenomenon.
result All relevant loss functions (CE, LS, FL, MSE) produce equivalent features and similar performance.
This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.
AGOP mechanism explains deep neural collapse in neural networks.
problem Explaining the rigid structure of data representations in deep neural networks.
method Introducing AGOP and Deep RFM to demonstrate DNC.
result AGOP mechanism causes deep neural collapse in neural networks.
GANs mode collapse solved with Bures distance.
problem GANs mode collapse or mode dropping.
method Use Bures distance to match real and fake batch diversity in feature space.
result Diversity matching reduces mode collapse and improves sample quality.
InfoQGAN uses mutual information to improve QGANs, overcoming mode collapse and feature disentanglement issues.
problem Mode collapse and lack of feature control in QGANs.
method Integrates InfoGAN principles with variational quantum circuit, classical discriminator, and MINE for mutual information optimization.
result InfoQGAN effectively mitigates mode collapse and achieves robust feature disentanglement.
Proposes PSCs for UQ in deep nets without retraining.
problem Estimating uncertainty in deep nets with a single pass.
method Identifies sensitive, smooth intermediate layer, fits probabilistic model.
result PSCs achieve UQ and OOD detection performance matching existing methods.
Variational autoencoders (VAEs) hold great potential for modelling text, as they could in theory separate high-level semantic and syntactic properties from local regularities of natural language. Practically, however, VAEs with autoregressive decoders often suffer from posterior collapse, a phenomenon where the model l…
Our research proves neural collapse in deep ResNets and transformers is globally optimal.
problem Understanding neural collapse in deep learning models.
method Analysis of deep regularized transformers and ResNets trained with cross entropy or mean squared error loss.
result Global optima of deep regularized transformers and ResNets are approximately collapsed, becoming more prominent as depth increases.
The paper tackles model collapse in GPLVMs by improving kernel flexibility and projection variance.
problem Model collapse in GPLVMs leading to vague latent representations.
method Theoretical analysis of projection variance, integration of SM and RFF kernels, and variational inference.
result The advisedRFLVM outperforms competing models in informative latent representations and missing data imputation.
Generative Adversarial Networks are known for their high quality outputs and versatility. However, they also suffer the mode collapse in their output data distribution. There have been many efforts to revamp GANs model and reduce mode collapse. This paper focuses on two of these models, PacGAN and VEEGAN. This paper ex…
Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. Yet, the interplay between graph topology and feature evolution in GNNs is not well understood. In this paper, we focus on node-wise classification, illustrated with community detection on stochastic block m…
DVE uses GPs on DNN outputs to provide UQ without retraining.
problem Feature collapse in DNNs affects UQ methods.
method Deep Vecchia ensemble (DVE) of GPs on DNN hidden layers.
result Deterministic UQ possible in feature-collapsed DNNs.
This paper explores how kernel methods can explain data effects on neural collapse.
problem Understanding how data affects neural collapse in neural networks.
method Formulating NC1 as a function of kernel, specializing to NNGP and NTK, and exploring a data-aware Gaussian Process kernel.
result The NTK does not represent more collapsed features than the NNGP for Gaussian data, highlighting the limitations of data-independent kernels.
Study shows neural collapse is invariant to class imbalances under certain conditions.
problem Neural collapse properties are only valid for balanced data.
method Adopted UFM and introduced SELI for invariant characterization.
result Embeddings and classifiers always interpolate a simplex-encoded label matrix regardless of class imbalances.
Study explains how noisyGD with DP improves feature learning despite high dimensionality.
problem Improving feature learning in differential privacy settings with noisyGD.
method Layer-peeled model in representation learning, error bound analysis, feature normalization, PCA.
result Misclassification error is independent of dimension in NC, and PCA improves testing accuracy.
Einstein metrics are blocked by manifold features and group growth.
problem Existence of Einstein metrics on specific 4-manifolds.
method Analysis of collapsing and group growth effects.
result Several 4-manifolds cannot support Einstein metrics due to specific features.
New findings show DNC is not optimal for deep models, revealing a low-rank bias.
problem Theoretical limitations of DNC in non-linear models and multi-class classification.
method Analysis of non-linear models of arbitrary depth in multi-class classification.
result DNC stops being optimal for DUFM when going beyond two layers or two classes, due to a low-rank bias.
A new loss function HUG decouples and generalizes neural collapse.
problem Neural collapse limits in deep learning models.
method Hyperspherical uniformity gap (HUG) as a unified framework.
result HUG decouples and generalizes neural collapse, improving model flexibility and robustness.
We address feature interpretation and reproducibility issues in dense nets, proposing a modified loss function.
problem Feature interpretation and reproducibility issues in dense nets.
method Proposed a modified loss function to circumvent basis collapse.
result Substantially concise nets with 100x fewer parameters and lower MSE loss.
Simplified non-contrastive learning avoids representation collapse.
problem Training failure modes in self-supervised learning.
method Hyperdimensional computing and inductive bias.
result The approach avoids representation collapses.
Paper explains neural collapse in neural networks using a new model.
problem Understanding neural collapse in neural networks during training.
method Introducing the unconstrained layer-peeled model (ULPM) to prove gradient flow convergence to critical points of a minimum-norm separation problem.
result Proves that all critical points are strict saddle points except the global minimizers exhibiting neural collapse.
Study feature representations induced by dependence between variables.
problem Learning feature representations from dependent random variables.
method Characterized sufficient and necessary conditions for dependence-induced representations, and provided a family of loss functions.
result Features learned from the family of loss functions can be expressed as the composition of a loss-dependent function and the maximal correlation function.
Indian Buffet Process based models are an elegant way for discovering underlying features within a data set, but inference in such models can be slow. Inferring underlying features using Markov chain Monte Carlo either relies on an uncollapsed representation, which leads to poor mixing, or on a collapsed representation…
This work explains neural collapse in shallow neural networks and its impact on generalization.
problem Understanding neural collapse in shallow neural networks and its effect on generalization.
method Analysis of two and three-layer ReLU neural networks, focusing on data dimension, sample size, and signal-to-noise ratio.
result Neural collapse occurs in shallow ReLU networks under certain conditions related to data properties and network architecture.
Gradient descent on LSE objectives implicitly performs EM, leading to collapse without volume control.
problem Gradient collapse in autoencoders without volume control.
method Introduced a single-layer encoder with an LSE objective and InfoMax regularization for volume control.
result Gradient--responsibility identity holds exactly; LSE alone collapses; variance prevents dead components; decorrelation prevents redundancy.
Elliptical Attention improves transformer performance by focusing on contextually relevant features.
problem Transformer models suffer from representation collapse and are vulnerable to contaminated samples.
method Uses Mahalanobis distance to define hyper-ellipsoidal neighborhoods for attention weights.
result Elliptical Attention reduces representation collapse and enhances model robustness.
Anomaly detection algorithms find extensive use in various fields. This area of research has recently made great advances thanks to deep learning. A recent method, the deep Support Vector Data Description (deep SVDD), which is inspired by the classic kernel-based Support Vector Data Description (SVDD), is capable of si…
Introduce Collapsed Effective Operators for higher-order structures.
problem Existing spectral operators decompose topology into separate ranks, leaving practitioners to fuse information back to vertices.
method Introduce Collapsed Effective Operators via Schur complementation of a graded Laplacian.
result Preserves positive semi-definiteness, lowers system energy under higher-order connectivity.
Transformers without skip connections collapse token representations to a single direction.
problem Rapid convergence of token representations to a single direction in self-attention-only Transformers.
method Analysis of layer normalization, residual connections, and multi-head attention mechanisms.
result Residual connections prevent rank collapse in real Transformers, while MLPs generate new feature directions.
Paper proposes a new stock price forecasting method using DRAGAN and feature matching.
problem Capturing correlations and training instability in GANs for stock price forecasting.
method Introduces DRAGAN and feature matching for improved training stability and correlation capture.
result Proposed method outperforms LSTM and basic GANs in stock price forecasting.
Study reveals LLM personas have two distinct components: frame-robust aggregated traits and frame-dependent geometric features.
problem Evaluation of LLM personas via psychometric questionnaires discards within-instance correlation structure.
method Constructed within-instance correlation matrices from IPIP-50 responses and analyzed geometry on SPD manifolds under manipulated question orderings.
result Persona expression comprises two dissociable components: aggregated features (Big Five scores) and geometric features (SPD manifold).
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.