Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
problem Understanding and optimizing deep learning training phases.
method Direct measurements on three deepnet architectures across seven datasets.
result Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
Contrastive learning struggles with class collapse and feature suppression, revealing bias towards simpler solutions.
problem Contrastive learning struggles with class collapse and feature suppression, especially in supervised and unsupervised settings.
method Unified theoretical framework to determine which features are learnt by CL, revealing bias towards simpler solutions.
result Bias towards simpler solutions is a key factor in class collapse and feature suppression.
This paper explains a mechanism called phase collapse that improves image classification accuracy.
problem Understanding the role of non-linearities and convolutional filters in image classification.
method Demonstrates phase collapse as a mechanism that eliminates spatial variability and linearly separates classes.
result Phase collapse improves classification accuracy, while thresholding operators degrade performance.
New findings show DNC is not optimal for deep models, revealing a low-rank bias.
problem Theoretical limitations of DNC in non-linear models and multi-class classification.
method Analysis of non-linear models of arbitrary depth in multi-class classification.
result DNC stops being optimal for DUFM when going beyond two layers or two classes, due to a low-rank bias.
This paper extends neural collapse to class-imbalanced datasets using an unconstrained ReLU feature model.
problem Understanding neural collapse in class-imbalanced datasets with cross-entropy loss.
method Generalized neural collapse to class-imbalanced settings using an unconstrained ReLU feature model.
result Class-means converge to orthogonal vectors with different lengths, and classifier weights align to these vectors.
We present a new approach to simple homotopy theory of polyhedra using finite topological spaces. We define the concept of collapse of a finite space and prove that this new notion corresponds exactly to the concept of a simplicial collapse. More precisely, we show that a collapse of finite spaces induces a simplicial …
Study shows neural collapse is invariant to class imbalances under certain conditions.
problem Neural collapse properties are only valid for balanced data.
method Adopted UFM and introduced SELI for invariant characterization.
result Embeddings and classifiers always interpolate a simplex-encoded label matrix regardless of class imbalances.
Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.
problem Understanding Neural Collapse in MSE-trained deep nets.
method Developed a new MSE loss decomposition and introduced the central path concept.
result Exact dynamics of Neural Collapse along the central path can be predicted.
We provide an algebraic description of the Teichmüller space and moduli space of flat metrics on a closed manifold or orbifold and study its boundary, which consists of (isometry classes of) flat orbifolds to which the original object may collapse. It is also shown that every closed flat orbifold can be obtained by col…
A new loss function HUG decouples and generalizes neural collapse.
problem Neural collapse limits in deep learning models.
method Hyperspherical uniformity gap (HUG) as a unified framework.
result HUG decouples and generalizes neural collapse, improving model flexibility and robustness.
Neural collapse occurs in normalized features over a Riemannian manifold.
problem Understanding neural collapse in normalized feature models.
method Simplified multi-class classification task to a nonconvex optimization problem over the Riemannian manifold, analyzing the landscape of critical points.
result The only global minimizers are neural collapse solutions, with all other critical points being strict saddles.
New model explains neural collapse and limits on minority classes in imbalanced datasets.
problem Understanding and predicting performance limits of deep learning models on imbalanced datasets.
method Layer-Peeled Model, a nonconvex optimization program isolating top layers and applying constraints.
result Reveals a new phenomenon called Minority Collapse that limits deep learning models on minority classes.
It was recently proved that embedded solutions of Euclidean hypersurface flows with speeds given by concave (convex), degree one homogeneous functions of the Weingarten map are interior (exterior) non-collapsing. These results were subsequently extended to hypersurface flows in the sphere and hyperbolic space. In the f…
Variational autoencoders often collapse, showing latent variables are non-identifiable.
problem Posterior collapse in variational autoencoders due to non-identifiable latent variables.
method Proves latent variable non-identifiability causes posterior collapse. Proposes latent-identifiable models using Brenier maps and input convex neural networks.
result Latent-identifiable models resolve posterior collapse and provide meaningful representations.
New analysis shows how attention masks and LayerNorm prevent rank collapse in transformers.
problem Rank collapse in transformer models with increasing depth.
method General analysis of rank collapse under self-attention, considering attention masks and LayerNorm.
result Self-attention with LayerNorm can prevent rank collapse and maintain a rich set of equilibria.
Study shows 'Ordinal Neural Collapse' in deep OR tasks, revealing simple geometric relationships.
problem Understanding neural collapse in deep Ordinal Regression tasks.
method Combining cumulative link models and Unconstrained Feature Model to investigate neural collapse.
result Demonstrates 'Ordinal Neural Collapse' (ONC) with three key properties.
This paper extends neural collapse to imbalanced data under cross-entropy loss.
problem Analyzing neural collapse in deep networks with imbalanced data.
method Using the unconstrained feature model and cross-entropy loss, the paper studies neural collapse in imbalanced datasets.
result Feature vectors within the same class collapse to a single mean vector, but angles between them depend on sample size.
We introduce the theory of strong homotopy types of simplicial complexes. Similarly to classical simple homotopy theory, the strong homotopy types can be described by elementary moves. An elementary move in this setting is called a strong collapse and it is a particular kind of simplicial collapse. The advantage of usi…
We analyze the topology and geometry of a polyhedron of dimension 2 according to the minimum size of a cover by PL collapsible polyhedra. We provide partial characterizations of the polyhedra of dimension 2 that can be decomposed as the union of two PL collapsible subpolyhedra in terms of their simple homotopy type and…
Uniform volume estimate for Kähler metrics in big cohomology classes.
problem Estimating volume for singular Kähler metrics in big cohomology classes.
method Generalized mixed energy estimate for functions in complex Sobolev space to big cohomology classes.
result Uniform non-collapsing volume estimate for local Kähler metrics.
We show that if on a compact Kahler threefold there is a solution of the Kahler-Ricci flow which encounters a finite time collapsing singularity, then the manifold admits a Fano fibration. Furthermore, if there is finite time extinction then the manifold is Fano and the initial class is a positive multiple of the first…
Generalizes Anosov flows to partially hyperbolic diffeomorphisms.
problem Classifying partially hyperbolic diffeomorphisms.
method Introducing collapsed Anosov flows and self orbit equivalences.
result All examples in Bonatti et al. belong to the collapsed Anosov flow class.
Wide neural networks with weight decay exhibit neural collapse.
problem Proving neural collapse in wide neural networks trained with weight decay.
method Generic guarantees on neural collapse for wide networks with weight decay, proving low training error and balancedness, and bounded conditioning.
result First proof of neural collapse in end-to-end training of wide neural networks with weight decay.
We consider the geometric inverse problem of determining a closed Riemannian manifold from measurements of the heat kernel in an open subset of the manifold. In this paper we analyze the stability of this problem in the class of n-dimensional Riemannian manifolds with bounded diameter and sectional curvature. It is w…
New method prevents class collapse in metric learning with margin-based losses.
problem Class collapse in metric learning due to diverse intra-class samples.
method Proposed a sampling method to select nearest same-class samples as positive elements in tuple.
result Demonstrated clear benefits on various fine-grained image retrieval datasets.
This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.
Deep linear networks exhibit collapsing features and classifiers across datasets.
problem Understanding the collapse of features and classifiers in deep linear networks.
method Theoretical and empirical analysis of deep linear networks with MSE and CE losses.
result Deep linear networks exhibit NC properties, collapsing features and classifiers to orthogonal vectors.
Deep learning predicts nuclear equation of state from rotating core collapse GW signals.
problem Classifying the nuclear equation of state from rotating core collapse gravitational wave signals.
method Employed deep convolutional neural networks to classify visual and temporal patterns in GW signals.
result Up to 97% correct classifications of nuclear equation of state in the test set.
New principles for collapsing law-invariant functionals to means, extending beyond convexity.
problem Conditions for law-invariant functionals to reduce to means.
method Establishing collapse to the mean principles for non-convex functionals.
result General principles apply beyond convexity, including quasiconvex and Choquet integrals.
Paper proves collapsing result for orbifolds without curvature bounds.
problem Proving collapsing result for orbifolds without curvature bounds.
method Introduces weak submersions and stratified Riemannian metrics.
result Allows Gromov-Hausdorff limits of orbifolds with strictly lower dimension.
We analyze neural collapse in neural networks, showing that features collapse to vertices of a Simplex ETF.
problem Understanding and optimizing the features learned in the last layer of neural networks during training.
method Simplified unconstrained feature model, studying the global optimization landscape of cross-entropy loss with weight decay.
result The global minimizers of the loss are Simplex ETFs, and other critical points are strict saddles with negative curvature.
Topic models, and more specifically the class of Latent Dirichlet Allocation (LDA), are widely used for probabilistic modeling of text. MCMC sampling from the posterior distribution is typically performed using a collapsed Gibbs sampler. We propose a parallel sparse partially collapsed Gibbs sampler and compare its spe…
Study on energy of maps from K3 surface to flat orbifold.
problem Energy of maps from K3 surface to flat orbifold.
method Investigate Dirichlet energy of smooth maps and introduce an invariant.
result Ratio of energy to invariant converges to 1 for Foscolo's collapsing families.
Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples. Despite remarkable recent improvements in generating realistic images, one of their major shortcomings is the fact that in practice, they tend to produce samples with little divers…
Durhuus and Jonsson (1995) introduced the class of "locally constructible" (LC) triangulated manifolds and showed that all the LC 2- and 3-manifolds are spheres. We show here that for each d>3 some LC d-manifolds are not spheres. We prove this result by studying how to collapse products of manifolds with exactly one fa…
This paper examines how different loss functions affect neural network features and performance.
problem Investigating which loss function is best for deep neural networks.
method Examining last-layer features of deep networks and drawing inspiration from the Neural Collapse phenomenon.
result All relevant loss functions (CE, LS, FL, MSE) produce equivalent features and similar performance.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
This work introduces a novel system for the generation of images that contain multiple classes of objects. Recent work in Generative Adversarial Networks have produced high quality images, but many focus on generating images of a single object or set of objects. Our system addresses the task of image generation conditi…
This work tackles posterior collapse in conditional and hierarchical VAEs.
problem Posterior collapse in VAEs leads to poor latent variable representations.
method Theoretical analysis of linear conditional and hierarchical VAEs, empirical validation.
result Theoretical and empirical evidence of posterior collapse causes in conditional and hierarchical VAEs.
The paper explores embedding Ricci flow solutions in flag manifolds.
problem Realizing Ricci flow solutions as embedded submanifolds.
method Investigation of invariant metrics in flag manifolds, proving global attractors and non-realizable collapses.
result Certain Ricci flow collapses cannot be embedded in Euclidean spaces.
Study of geometric actions on CAT(0) spaces and their limits.
problem Understanding limits of geometric actions on CAT(0) spaces.
method Analysis of convergence and splitting/collapsing phenomena in CAT(0) lattices and orbispaces.
result Proof of compactness theorem for CAT(0) homology orbifolds.
This paper investigates how large language models achieve neural collapse, a phenomenon linked to generalization.
problem Neural collapse in large language models under imbalanced and token-rich conditions.
method Empirical investigation of scaling and regularization effects on CLMs' progression towards neural collapse.
result Neural collapse properties develop with scale and regularization, linked to generalization in language modeling.
Theoretical work on mode collapse in variational inference models.
problem Mode collapse in variational inference models, where models focus on a few modes instead of all possible ones.
method Theoretical investigation of mode collapse in Gaussian mixture models, identifying key low-dimensional statistics and equations governing their evolution.
result Mode collapse is present even in favorable scenarios, driven by mean alignment and vanishing weight mechanisms.
Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. Yet, the interplay between graph topology and feature evolution in GNNs is not well understood. In this paper, we focus on node-wise classification, illustrated with community detection on stochastic block m…
Uniform entropy bound for Ricci shrinkers with bounded curvature.
problem Bounding entropy for Ricci shrinkers with specific curvature constraints.
method Establishing uniform entropy bounds for simply connected Ricci shrinkers with a finite second homotopy group and uniform curvature bounds.
result Uniform entropy bound for simply connected Ricci shrinkers with a finite second homotopy group and uniform curvature bounds.
Variational Bayesian inference and (collapsed) Gibbs sampling are the two important classes of inference algorithms for Bayesian networks. Both have their advantages and disadvantages: collapsed Gibbs sampling is unbiased but is also inefficient for large count values and requires averaging over many samples to reduce …
In this short note, exploits of constructions of F-structures coupled with technology developed by Cheeger-Gromov and Paternain-Petean are seen to yield a procedure to compute minimal entropy, minimal volume, Yamabe invariant and to study collapsing with bounded sectional curvature on inequivalent smooth st…
We study the collapsing behavior of the Kaehler-Ricci flow on a compact Kaehler manifold X admitting a holomorphic submersion X -> S coming from its canonical class, where S is a Kaehler manifold with dim S < dim X. We show that the flow metric degenerates at exactly the rate of e^{-t} as predicted by the cohomology in…