Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
problem Understanding and optimizing deep learning training phases.
method Direct measurements on three deepnet architectures across seven datasets.
result Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
Layer normalization with activations prevents Gram matrix rank collapse at initialization.
problem Rank collapse in Gram matrices at initialization slows training in deep networks.
method Proved that layer normalization, with activation layers, biases Gram matrix towards identity matrix at exponential rate.
result Layer normalization with activations biases Gram matrix towards identity matrix at exponential rate with depth at initialization.
AI task delegation faces incentive collapse with unbounded payments as AI accuracy rises.
problem Incentive collapse in AI-assisted task delegation schemes.
method General impossibility result and sentinel-auditing payment mechanism.
result Sentinel-auditing mechanism enforces positive human effort at finite cost, independent of AI accuracy.
Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show th…
REPAIR mitigates variance collapse to enable linear interpolation between SGD solutions.
problem Linear interpolation between SGD solutions is difficult due to variance collapse in permuted activations.
method REPAIR rescales preactivations of interpolated networks to mitigate variance collapse.
result 60%-100% relative barrier reduction across various architectures and tasks.
The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.
problem Oversmoothing and correlation collapse in deep ReLU GNNs.
method The paper derives and validates three principles for initialization and architecture selection in finite width graph neural networks with ReLU activations.
result Correct initialization, residual aggregation operators, and residual connections significantly improve early training dynamics in deep ReLU GNNs.
In image classification tasks, the ability of deep CNNs to deal with complex image data has proven to be unrivalled. However, they require large amounts of labeled training data to reach their full potential. In specialised domains such as healthcare, labeled data can be difficult and expensive to obtain. Active Learni…
NE-VAE prevents posterior collapse in VAEs by embedding neighbors in latent space.
problem Posterior collapse in VAEs when strong decoders are used.
method Neighbor embedding in latent space to prevent collapse.
result NE-VAE produces qualitatively different latent representations with active latent dimensions.
This work tackles posterior collapse in conditional and hierarchical VAEs.
problem Posterior collapse in VAEs leads to poor latent variable representations.
method Theoretical analysis of linear conditional and hierarchical VAEs, empirical validation.
result Theoretical and empirical evidence of posterior collapse causes in conditional and hierarchical VAEs.
This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.
Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.
problem Understanding Neural Collapse in MSE-trained deep nets.
method Developed a new MSE loss decomposition and introduced the central path concept.
result Exact dynamics of Neural Collapse along the central path can be predicted.
Geometric theory of projection heads in self-supervised learning.
problem Dimensional collapse and information invariance trade-off in projection heads.
method Geometric modeling of projection heads as Riemannian metrics, analyzing Hessian eigenvalues, and tracking optimization geometry.
result Smooth nonlinear heads induce negative curvature, preventing collapse; linear and ReLU heads cannot.
A new method normalizes activations to match batch normalization without batch dependence.
problem Performance degradation with batch-independent normalization techniques.
method Proxy-Normalizing Activations
result Proxy-Normalization technique emulates batch normalization's behavior and performance.
FTX's failure linked to Terra-Luna collapse and Binance's influence.
problem FTX's collapse due to misuse of native token and reliance on leverage.
method Analyzed on-chain data, studied cryptocurrency dependency structures, and examined public trades.
result FTX's downfall was accelerated by Binance's tweets and public reaction.
We analyze neural collapse in neural networks, showing that features collapse to vertices of a Simplex ETF.
problem Understanding and optimizing the features learned in the last layer of neural networks during training.
method Simplified unconstrained feature model, studying the global optimization landscape of cross-entropy loss with weight decay.
result The global minimizers of the loss are Simplex ETFs, and other critical points are strict saddles with negative curvature.
The paper explains how continuous language models can produce discrete, interpretable meanings.
problem Semantic collapse in continuous systems of large language models.
method Formalizing large language models as Continuous State Machines (CSMs) and analyzing the associated transfer operator.
result The leading eigenfunctions of the transfer operator induce a finite number of invariant meaning basins, explaining how continuous computation can produce discrete, interpretable semantics.
We investigate a multi-household DSGE model in which past aggregate consumption impacts the confidence, and therefore consumption propensity, of individual households. We find that such a minimal setup is extremely rich, and leads to a variety of realistic output dynamics: high output with no crises; high output with i…
Improving sparsity of deep neural networks (DNNs) is essential for network compression and has drawn much attention. In this work, we disclose a harmful sparsifying process called filter collapse, which is common in DNNs with batch normalization (BN) and rectified linear activation functions (e.g. ReLU, Leaky ReLU). It…
We demonstrate that the tail dependence should always be taken into account as a proxy for systematic risk of loss for investments. We provide the clear statistical evidence of that the structure of investment portfolios on a regulated market should be adjusted to the price of gold. Our finding suggests that the active…
This paper explores how kernel methods can explain data effects on neural collapse.
problem Understanding how data affects neural collapse in neural networks.
method Formulating NC1 as a function of kernel, specializing to NNGP and NTK, and exploring a data-aware Gaussian Process kernel.
result The NTK does not represent more collapsed features than the NNGP for Gaussian data, highlighting the limitations of data-independent kernels.
Active-GRPO improves molecular optimization by actively deciding when to imitate or self-improve.
problem Training robust and efficient molecular optimization models with large language models.
method Active-GRPO combines imitation and reinforcement learning, upgrading references and policies dynamically.
result Improves molecular optimization performance, achieving statistically significant gains.
We address feature interpretation and reproducibility issues in dense nets, proposing a modified loss function.
problem Feature interpretation and reproducibility issues in dense nets.
method Proposed a modified loss function to circumvent basis collapse.
result Substantially concise nets with 100x fewer parameters and lower MSE loss.
In recent years, there has seen much interest and increased research activities on Perelman's paper. Section one and two of this paper aim to establish Perelman's local non-collapsing result for the Ricci flow. This will provide a positive lower bound on the injectivity radius for the Ricci flow under blow-up analysis.…
A new mutual information lower bound for multimodal regression active learning.
problem Lack of effective acquisition functions for multimodal regression active learning.
method Introduces a Two-Index framework for separating epistemic and aleatoric sources of uncertainty, deriving MI-LB as a closed-form approximation.
result MI-LB consistently outperforms baselines on multimodal regression tasks.
Bayesian quadrature (BQ) is a sample-efficient probabilistic numerical method to solve integrals of expensive-to-evaluate black-box functions, yet so far,active BQ learning schemes focus merely on the integrand itself as information source, and do not allow for information transfer from cheaper, related functions. Here…
Bayesian deep learning faces posterior collapse due to likelihood vs. prior competition.
problem Posterior collapse in Bayesian deep learning models.
method Identified competition between likelihood and prior regularization in a linear latent variable model.
result Posterior collapse is related to neural and dimensional collapse, suggesting a broader learning issue.
Proves weakly non-collapsed RCD spaces are strongly non-collapsed.
problem Proving the equivalence of weakly non-collapsed and strongly non-collapsed RCD spaces.
method Analyzes properties of RCD spaces and uses auxiliary results.
result Confirms conjecture about RCD spaces being strongly non-collapsed.
Study on Neural Collapse limits in deep learning.
problem Understanding the limits of Neural Collapse in deep learning.
method Investigated Neural Collapse in the context of generalization and feature learning, refining conjectures and conducting experiments.
result Neural Collapse primarily occurs on the train set and not on the test set, suggesting it is an optimization phenomenon with unclear connections to generalization.
FF algorithm uses goodness as a measure of input quality, derived from likelihood-ratio tests.
problem Training each layer locally with a goodness measure.
method FF algorithm uses a likelihood-ratio test to define goodness, which is the sum of squared activations normalized between layers.
result The goodness measure is a sufficient statistic for a likelihood-ratio test, explaining the FF algorithm's performance.
Novel anti-grokking phase discovered in neural networks, revealed by HTSR layer quality metric.
problem Understanding and detecting overfitting in neural networks.
method 3-layer MLP, weight decay, HTSR layer quality metric α, correlation traps, Kolmogorov–Smirnov tests. result Anti-grokking phase detected late in training, revealed by α<2 and correlation traps. Dropout schedules can be optimized to significantly reduce model test loss.
problem Improving model performance in neural networks.
method Developed a mean-field theory of dropout at the edge of chaos, proposing front-loaded dropout schedules.
result Front-loaded dropout schedules reduce test loss by 18-35% over constant dropout.
Ricci flow smooths locally collapsing manifolds with controlled curvature.
problem Locally collapsing manifolds with controlled Ricci curvature.
method Ricci flow for a definite period of time, detecting collapsing infranil fiber bundles.
result Topological conditions detect collapsing infranil fiber bundles.
A framework connects VAEs to GLMs for better model initialization and performance.
problem Understanding and optimizing loss function critical points in VAEs.
method Introducing a theoretical framework based on GLM and EDFs.
result Maximum likelihood initialization improves VAE performance.
Mean representations of VAEs are correlated but still useful for tasks.
problem Correlation between mean and sampled representations of VAEs.
method Selective posterior collapse to identify active and passive variables.
result Passive variables in mean representations are correlated but uncorrelated in sampled ones.
Mathematical analysis shows annealing prevents mode collapse in Gaussian mixtures.
problem Mode collapse in variational inference for multimodal distributions.
method Analyzed annealing strategies for Gaussian mixtures, derived formulas, and tested on neural networks.
result Appropriately chosen annealing schemes can robustly prevent mode collapse.
The study characterizes and rules out collapsing in convex ancient mean curvature flow.
problem Characterizing and ruling out collapsing in convex ancient mean curvature flow.
method Characterization and counterexamples.
result Collapsing occurs if and only if the flow is asymptotic to at least one Grim hyperplane.
Despite excellent progress in recent years, mode collapse remains a major unsolved problem in generative adversarial networks (GANs).In this paper, we present spectral regularization for GANs (SR-GANs), a new and robust method for combating the mode collapse problem in GANs. Theoretical analysis shows that the optimal …
New method controls posterior collapse in VAEs without network architecture constraints.
problem Posterior collapse in VAEs reduces diversity of generated samples.
method Introduces Latent Reconstruction (LR) loss to control posterior collapse.
result Controls posterior collapse on various datasets without architectural constraints.
Special Lagrangian submanifolds emerge from K3 surface collapse.
problem Understanding special Lagrangian submanifolds in K3 surface collapse.
method Lifting affine lines to degenerating sequences of special Lagrangian submanifolds.
result Constructing special Lagrangian two-spheres connecting Taub-NUT bubbles.
Collapsibility is a combinatorial strengthening of contractibility. We relate this property to metric geometry by proving the collapsibility of any complex that is CAT(0) with a metric for which all vertex stars are convex. This strengthens and generalizes a result by Crowley. Further consequences of our work are: (1) …
Two-dimensional collapsed spaces with lower Ricci bounds are topological surfaces.
problem Topology of collapsed spaces with lower Ricci bounds
method Prove that collapsed spaces are topological surfaces
result Collapsed spaces are topological surfaces
Prove that collapsing CSC metrics can be perturbed to invariant collapsing CSC metrics.
problem Prove that collapsing constant scalar curvature metrics can be perturbed to invariant collapsing constant scalar curvature metrics.
method Prove that a sequence of constant scalar curvature metrics which is collapsing with bounded curvature to a manifold can be perturbed to a sequence of invariant collapsing constant scalar curvature metrics.
result Prove that a sequence of constant scalar curvature metrics which is collapsing with bounded curvature to a manifold can be perturbed to a sequence of invariant collapsing constant scalar curvature metrics.
We will simplify the earlier proofs of Perelman's collapsing theorem of 3-manifolds given by Shioya-Yamaguchi and Morgan-Tian. Among other things, we use Perelman's semi-convex analysis of distance functions to construct the desired local Seifert fibration structure on collapsed 3-manifolds. The verification of Perelma…
This paper finds ReLU restores symmetry in SCL under class imbalances.
problem Symmetry break in SCL under class imbalances.
method Analytical proof and experiments with ReLU activation and batch selection.
result ReLU restores symmetry in SCL-learned representations without loss in test accuracy.
Estimate collapsibility of causal effects in CPDAGs via strong d-convex hulls.
problem Estimate causal effects in CPDAGs.
method Use strong d-convex hulls to characterize minimal collapsible sets.
result Efficient algorithm for obtaining collapsible sets in DAGs and CPDAGs.
We introduce the theory of strong homotopy types of simplicial complexes. Similarly to classical simple homotopy theory, the strong homotopy types can be described by elementary moves. An elementary move in this setting is called a strong collapse and it is a particular kind of simplicial collapse. The advantage of usi…
Lower Ricci curvature bound prevents first Betti number from dropping more than dimension in collapsing manifolds.
problem Understanding how the first Betti number behaves under manifold collapse with Ricci curvature bounds.
method Analyzing sequences of Riemannian manifolds with lower Ricci curvature bounds.
result The first Betti number cannot drop more than the dimension in collapsing manifolds.
Study tackles criterion collapse in learning criteria, showing conditions for loss minimization.
problem Criterion collapse in optimization, focusing on error probability minimizers.
method Analyzes various learning criteria, including DRO, OCE risks, and non-monotonic criteria.
result Non-monotonic criteria can avoid collapse, while monotonic ones cannot.