This paper explains a mechanism called phase collapse that improves image classification accuracy.
problem Understanding the role of non-linearities and convolutional filters in image classification.
method Demonstrates phase collapse as a mechanism that eliminates spatial variability and linearly separates classes.
result Phase collapse improves classification accuracy, while thresholding operators degrade performance.
Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
problem Understanding and optimizing deep learning training phases.
method Direct measurements on three deepnet architectures across seven datasets.
result Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
Novel anti-grokking phase discovered in neural networks, revealed by HTSR layer quality metric.
problem Understanding and detecting overfitting in neural networks.
method 3-layer MLP, weight decay, HTSR layer quality metric α, correlation traps, Kolmogorov–Smirnov tests. result Anti-grokking phase detected late in training, revealed by α<2 and correlation traps. The paper analyzes the dynamics of tokens in transformer models at moderate interaction levels.
problem Understanding the evolution of tokens in transformer models at moderate interaction levels.
method Modeling transformer models as a system of particles interacting in a mean-field way and studying the corresponding dynamics.
result Characterization and convergence of the limiting dynamics in different phases of the system.
Trust is a collective, self-fulfilling phenomenon that suggests analogies with phase transitions. We introduce a stylized model for the build-up and collapse of trust in networks, which generically displays a first order transition. The basic assumption of our model is that whereas trust begets trust, panic also begets…
Paper explains neural collapse in neural networks using a new model.
problem Understanding neural collapse in neural networks during training.
method Introducing the unconstrained layer-peeled model (ULPM) to prove gradient flow convergence to critical points of a minimum-norm separation problem.
result Proves that all critical points are strict saddle points except the global minimizers exhibiting neural collapse.
Inspired by the recent literature on aggregation theory, we aim at relating the long range correlation of the stocks return volatility to the heterogeneity of the investors' expectations about the level of the future volatility. Based on a semi-parametric model of investors' anticipations, we make the connection betwee…
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.
problem Reward collapse in aligning large language models with human preferences.
method Introduced a prompt-aware optimization scheme to derive closed-form expressions for reward distributions.
result Our prompt-aware utility functions significantly alleviate reward collapse during training.
Generative diffusion models gradually memorize training data, losing independent dimensions.
problem Understanding how generative diffusion models memorize training data, especially on low-dimensional manifolds.
method Measuring latent dimensionality via the learned score field, proposing a geometric memorization theory.
result Generative diffusion models experience a smooth collapse of their capacity to vary across independent directions as data become scarce, leading to near point-wise replication of salient features.
Study shows how anisotropic data affects learning dynamics in phase retrieval.
problem Understanding learning dynamics in phase retrieval with anisotropic Gaussian inputs.
method Developed a tractable reduction to reveal a three-phase trajectory and derived scaling laws.
result Found that anisotropy leads to a three-phase trajectory: fast escape, slow convergence, and spectral-tail learning.
An elementary family of local Hamiltonians H,¸ℓ,ℓ=1,2,3,ldots, is described for a 2−dimensional quantum mechanical system of spin =1/2 particles. On the torus, the ground state space G∘,ℓ is (log) extensively degenerate but should collapse under łperturbation" to an anyonic syste…
Improved RL training for DMs reduces mode collapse and preserves diversity.
problem Mode collapse and training instability in RL fine-tuned diffusion models.
method Dynamic hierarchical RL training with sliding-window parameter regularisation.
result Models trained with HRF achieve better preservation of diversity in downstream tasks.
Our paper explains deep neural collapse in multiple layers.
problem Understanding deep neural collapse in multi-layered neural networks.
method Generalized unconstrained features model for deep networks.
result Deep unconstrained features model exhibits deep neural collapse.
Given a set of points that sample a shape, the Rips complex of the data points is often used in machine-learning to provide an approximation of the shape easily-computed. It has been proved recently that the Rips complex captures the homotopy type of the shape assuming the vertices of the complex meet some mild samplin…
Modeling how network connectivity affects economic collapse and robustness.
problem Impact of network topology on systemic risk and collapse of complex economic systems.
method Proposed a model to study the effects of network structure on economic systems by varying connectivity.
result Emergent systemic risks arise with increased interconnections, leading to phase transitions and tipping points.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
A new approach uses circuit topology to study complex polymer interactions.
problem Understanding structural phase transitions in entangled polymer systems.
method Braided circuit topology framework for multiple-chain systems.
result Circuit topological motif fractions are effective order parameters for structural transitions.
This paper extends neural collapse to imbalanced data under cross-entropy loss.
problem Analyzing neural collapse in deep networks with imbalanced data.
method Using the unconstrained feature model and cross-entropy loss, the paper studies neural collapse in imbalanced datasets.
result Feature vectors within the same class collapse to a single mean vector, but angles between them depend on sample size.
This paper extends neural collapse to class-imbalanced datasets using an unconstrained ReLU feature model.
problem Understanding neural collapse in class-imbalanced datasets with cross-entropy loss.
method Generalized neural collapse to class-imbalanced settings using an unconstrained ReLU feature model.
result Class-means converge to orthogonal vectors with different lengths, and classifier weights align to these vectors.
Study on free boundary problems in RCD spaces, proving existence and regularity.
problem Free boundary problems in RCD metric measure spaces.
method Existence and local Lipschitz regularity of solutions, free boundary analysis.
result Existence and regularity of solutions, free boundary structure.
We analyze neural collapse in neural networks, showing that features collapse to vertices of a Simplex ETF.
problem Understanding and optimizing the features learned in the last layer of neural networks during training.
method Simplified unconstrained feature model, studying the global optimization landscape of cross-entropy loss with weight decay.
result The global minimizers of the loss are Simplex ETFs, and other critical points are strict saddles with negative curvature.
Persistent entropy detects phase transitions in complex systems.
problem Detecting phase transitions in complex systems.
method Established a general theorem for persistent entropy to reliably detect phase transitions, introduced operational framework for finite-time computations.
result Persistent entropy exhibits an asymptotically non-vanishing gap across phases, robust numerical signatures across experiments.
High-dimensional models become unstable when sample size falls below a critical level, leading to a phase transition.
problem Instability in high-dimensional learning models when sample size is insufficient.
method Proved the necessity of a Fisher eigenvalue threshold for stability, introduced Fisher floor for verification.
result A sharp phase transition between reliable concentration and inevitable failure in high-dimensional learning.
Study on line bundle flow on Kähler surfaces converging to a singular solution.
problem Analyzing the mean curvature flow on Kähler surfaces.
method Investigates the flow under hypercritical phase and semipositivity conditions.
result The flow converges to a singular solution away from curves of negative self-intersection.
For surfaces, we brush a reasonably sharp picture of the influence of the fundamental group upon the complexity of foliated-dynamics. A metaphor emerges with phase-changes through the solid-liquid-gaseous states. Groups of ranks 0≤r≤1 are frozen with intransitivity reigning ubiquitously. When 2≤r≤3, th…
New estimates quantify blow-up rates of spacelike singularities in gravitational collapse.
problem Quantifying the blow-up rates of spacelike singularities in gravitational collapse.
method Deriving new quantitative estimates using spherical symmetry and double-null coordinates.
result Polynomial blow-up rates O(1/rN) for various quantities, with improved estimates for r∂ur and r∂vr. Deep neural networks near edge of chaos show universal scaling laws.
problem Understanding the behavior of deep neural networks near critical points.
method Analogy to absorbing phase transitions in statistical mechanics, deterministic propagation dynamics, mean-field and directed percolation universality classes.
result Deep neural networks exhibit universal scaling laws near the edge of chaos.
Since the 2007-2009 financial crisis, substantial academic effort has been dedicated to improving our understanding of interbank lending networks (ILNs). Because of data limitations or by choice, the literature largely lacks multiple loan maturities. We employ a complete interbank loan contract dataset to investigate w…
A new generator uses kernel distance to avoid GAN weaknesses.
problem Stability and mode collapse in GANs and autoencoders.
method LCW generator (Latent Cramer-Wold generator) using kernel distance.
result Very competitive FID values.
This work explains neural collapse in shallow neural networks and its impact on generalization.
problem Understanding neural collapse in shallow neural networks and its effect on generalization.
method Analysis of two and three-layer ReLU neural networks, focusing on data dimension, sample size, and signal-to-noise ratio.
result Neural collapse occurs in shallow ReLU networks under certain conditions related to data properties and network architecture.
Theoretical study shows AI models can recover from contaminated training data.
problem Data contamination in AI training can degrade model performance.
method Theoretical analysis and experiments on various data types.
result Models converge to true distribution under mild conditions, with rate dependent on real data fraction.
The aim of this work is to explore the possible types of phenomena that simple macroeconomic Agent-Based models (ABM) can reproduce. We propose a methodology, inspired by statistical physics, that characterizes a model through its 'phase diagram' in the space of parameters. Our first motivation is to understand the lar…
Ancient solutions found on flag manifolds from invariant Einstein metrics.
problem Understanding the behavior of Ricci flow on flag manifolds.
method Global study of the dynamical system induced by the Ricci flow, using invariant Einstein metrics and Poincaré compactification.
result Non-collapsed ancient solutions emerge from invariant Einstein metrics, with a Type I singularity in finite time.
Proposes using continuum percolation to analyze data manifolds and improve generative models.
problem Disentangling geometric support from probability distributions in high-dimensional data.
method Establishes a correspondence between topological phase transitions of random geometric graphs and data manifolds, using Percolation Shift metric.
result Demonstrates that Percolation Shift metric captures structural pathologies like mode collapse and guides training to prevent manifold shrinkage and improve fidelity.
Bayesian deep learning faces posterior collapse due to likelihood vs. prior competition.
problem Posterior collapse in Bayesian deep learning models.
method Identified competition between likelihood and prior regularization in a linear latent variable model.
result Posterior collapse is related to neural and dimensional collapse, suggesting a broader learning issue.
Proves weakly non-collapsed RCD spaces are strongly non-collapsed.
problem Proving the equivalence of weakly non-collapsed and strongly non-collapsed RCD spaces.
method Analyzes properties of RCD spaces and uses auxiliary results.
result Confirms conjecture about RCD spaces being strongly non-collapsed.
Study on Neural Collapse limits in deep learning.
problem Understanding the limits of Neural Collapse in deep learning.
method Investigated Neural Collapse in the context of generalization and feature learning, refining conjectures and conducting experiments.
result Neural Collapse primarily occurs on the train set and not on the test set, suggesting it is an optimization phenomenon with unclear connections to generalization.
Novel Ricci flow normalization for homogeneous spaces, focusing on flag manifolds.
problem Understanding the limiting behavior and symmetry properties of Ricci flow on homogeneous spaces.
method Introducing a novel normalization for the homogeneous Ricci flow and characterizing Gromov-Hausdorff limits.
result Full classification of Gromov-Hausdorff limits and detailed phase portraits for three-isotropy-summands flag manifolds.
Ricci flow smooths locally collapsing manifolds with controlled curvature.
problem Locally collapsing manifolds with controlled Ricci curvature.
method Ricci flow for a definite period of time, detecting collapsing infranil fiber bundles.
result Topological conditions detect collapsing infranil fiber bundles.
Mathematical analysis shows annealing prevents mode collapse in Gaussian mixtures.
problem Mode collapse in variational inference for multimodal distributions.
method Analyzed annealing strategies for Gaussian mixtures, derived formulas, and tested on neural networks.
result Appropriately chosen annealing schemes can robustly prevent mode collapse.
The study characterizes and rules out collapsing in convex ancient mean curvature flow.
problem Characterizing and ruling out collapsing in convex ancient mean curvature flow.
method Characterization and counterexamples.
result Collapsing occurs if and only if the flow is asymptotic to at least one Grim hyperplane.
Despite excellent progress in recent years, mode collapse remains a major unsolved problem in generative adversarial networks (GANs).In this paper, we present spectral regularization for GANs (SR-GANs), a new and robust method for combating the mode collapse problem in GANs. Theoretical analysis shows that the optimal …
New method controls posterior collapse in VAEs without network architecture constraints.
problem Posterior collapse in VAEs reduces diversity of generated samples.
method Introduces Latent Reconstruction (LR) loss to control posterior collapse.
result Controls posterior collapse on various datasets without architectural constraints.
Study flat manifolds' collapsed limits as flat orbifolds.
problem Understanding collapsed limits of flat manifolds.
method Analyzing totally geodesic foliations and Gromov-Hausdorff limits.
result Identify collapsed limits as flat orbifolds and provide criteria for singularity.
Special Lagrangian submanifolds emerge from K3 surface collapse.
problem Understanding special Lagrangian submanifolds in K3 surface collapse.
method Lifting affine lines to degenerating sequences of special Lagrangian submanifolds.
result Constructing special Lagrangian two-spheres connecting Taub-NUT bubbles.
The torus cannot collapse to a segment under certain curvature conditions.
problem Impossibility of codimension-one collapse for surfaces of negative Euler characteristic.
method Analysis of Gaussian curvature and homology loops.
result The torus cannot collapse to a segment under similar conditions to surfaces of negative Euler characteristic.
Collapsibility is a combinatorial strengthening of contractibility. We relate this property to metric geometry by proving the collapsibility of any complex that is CAT(0) with a metric for which all vertex stars are convex. This strengthens and generalizes a result by Crowley. Further consequences of our work are: (1) …