Feature normalization prevents collapse in non-contrastive learning dynamics.
problem Non-contrastive learning can collapse into a single point due to lack of repulsive force.
method Extended previous theory based on L2 loss to cosine loss, considering feature normalization.
result Cosine loss induces stable equilibrium, preventing collapse even with insufficient repulsive force.
New model explains GAN training dynamics and mode collapse.
problem Mode collapse in GANs, where generators fail to reproduce diversity.
method Simplified model using particles in output space, coupled by universal kernel.
result Gradient regularizers can optimally yield convergence through critical damping.
Use simplified layerwise linear models to understand neural dynamics.
problem Complex neural network dynamics are hard to grasp.
method Apply simplified layerwise linear models to explain neural phenomena.
result Simplified models explain neural collapse, emergence, etc.
KL annealing helps VAEs avoid posterior collapse and overfitting.
problem Posterior collapse and overfitting in VAEs.
method Theoretical analysis of learning dynamics with KL annealing.
result Posterior collapse is inevitable when β exceeds a threshold. Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.
problem Understanding Neural Collapse in MSE-trained deep nets.
method Developed a new MSE loss decomposition and introduced the central path concept.
result Exact dynamics of Neural Collapse along the central path can be predicted.
Study shows partial hyperbolicity leads to Anosov dynamics in 3-manifolds.
problem Understanding dynamics in hyperbolic 3-manifolds and Seifert manifolds.
method Classification of partially hyperbolic diffeomorphisms and pseudo-Anosov dynamics.
result Complete classification of partially hyperbolic diffeomorphisms in hyperbolic 3-manifolds and Seifert manifolds.
Paper proves existence of anisotropic dynamical horizons in gravitational collapse.
problem Existence of apparent horizons in gravitational collapse.
method Scale-critical hyperbolic method and non-perturbative elliptic techniques.
result Smooth and spacelike apparent horizons emerge from general initial data in gravitational collapse.
New analysis shows how attention masks and LayerNorm prevent rank collapse in transformers.
problem Rank collapse in transformer models with increasing depth.
method General analysis of rank collapse under self-attention, considering attention masks and LayerNorm.
result Self-attention with LayerNorm can prevent rank collapse and maintain a rich set of equilibria.
We study the mean curvature flow of hypersurfaces in Rn+1, with initial surfaces sufficiently close to the standard n-dimensional sphere. The closeness is in the Sobolev norm with the index greater than 2n+1 and therefore it does not impose restrictions of the mean curvature of the initial surface. W…
Self-attention networks localize when eigenspectrum variance is small.
problem Self-attention mechanisms can lead to rank and entropy collapses, reducing model expressivity and trainability.
method Characterized attention localization using query-key eigenspectrum variance.
result Small eigenspectrum variance prevents both rank and entropy collapses, improving model performance.
This study investigates abrupt learning dynamics in Transformers, revealing plateau formation and internal representation collapse.
problem Abrupt learning in Transformers, particularly during the loss plateau.
method Investigates mechanisms of abrupt learning in shallow Transformers, focusing on attention maps and hidden states.
result Reveals plateau formation, internal representation collapse, and strong repetition bias in outputs.
Populations of species in ecosystems are often constrained by availability of resources within their environment. In effect this means that a growth of one population, needs to be balanced by comparable reduction in populations of others. In neutral models of biodiversity all populations are assumed to change increment…
Study finds significant price declines and capital reallocation from centralized to decentralized exchanges after FTX collapse.
problem Quantifying trust dynamics and redistribution between centralized and decentralized exchanges.
method Interdisciplinary approach combining causal inference and computational text analysis.
result Significant price declines and capital reallocation from centralized to decentralized exchanges following the FTX collapse.
New insights into CE dynamics reveal how Hadamard initialization simplifies softmax.
problem Understanding the dynamics of cross-entropy training loss in deep learning.
method Analyzing a two-layer linear neural network with standard-basis vectors as inputs.
result Gradient flow on cross-entropy converges to neural collapse geometry, proving global convergence.
New metric measures dynamical richness without relying on accuracy.
problem Lack of a reliable metric for measuring dynamical richness.
method Developed a computationally efficient, performance-independent metric based on low-rank bias.
result Metric recovers neural collapse as a special case and captures known transitions without accuracy.
We propose and analyze numerically a simple dynamical model that describes the firm behaviors under uncertainty of demand forecast. Iterating this simple model and varying some parameters values we observe a wide variety of market dynamics such as equilibria, periodic and chaotic behaviors. Interestingly the model is a…
Constructs initial data for multiple black holes with specified ADM parameters.
problem Forming multiple black holes with specific ADM parameters.
method Smooth, asymptotically flat vacuum initial data with prescribed ADM energy, momentum, and angular momentum.
result Maximal development of data results in spacetimes containing multiple black holes.
We propose to incorporate adversarial dropout in generative multi-adversarial networks, by omitting or dropping out, the feedback of each discriminator in the framework with some probability at the end of each batch. Our approach forces the single generator not to constrain its output to satisfy a single discriminator,…
Gradient descent on LSE objectives implicitly performs EM, leading to collapse without volume control.
problem Gradient collapse in autoencoders without volume control.
method Introduced a single-layer encoder with an LSE objective and InfoMax regularization for volume control.
result Gradient--responsibility identity holds exactly; LSE alone collapses; variance prevents dead components; decorrelation prevents redundancy.
Levenshtein VAE prevents posterior collapse in text generation models.
problem Posterior collapse in VAEs where generators ignore latent variables.
method Replaces ELBO with a Levenshtein distance-based objective to prevent collapse.
result Levenshtein VAE produces more informative latent representations.
The paper analyzes the dynamics of tokens in transformer models at moderate interaction levels.
problem Understanding the evolution of tokens in transformer models at moderate interaction levels.
method Modeling transformer models as a system of particles interacting in a mean-field way and studying the corresponding dynamics.
result Characterization and convergence of the limiting dynamics in different phases of the system.
Paper studies apparent horizon dynamics and introduces a null comparison principle.
problem Global dynamics of apparent horizon and local achronality.
method Constructing apparent horizon by solving MOTS along null hypersurfaces, using Klainerman-Szeftel estimates and null comparison principle.
result Smooth, asymptotically null, and converging apparent horizon proven.
The paper explains how continuous language models can produce discrete, interpretable meanings.
problem Semantic collapse in continuous systems of large language models.
method Formalizing large language models as Continuous State Machines (CSMs) and analyzing the associated transfer operator.
result The leading eigenfunctions of the transfer operator induce a finite number of invariant meaning basins, explaining how continuous computation can produce discrete, interpretable semantics.
A new method recovers latent potentials from graph flows, preserving ordering and stability.
problem Recovering latent potentials from graph flows is ill-posed and standard methods collapse the ordering.
method Gauge-invariant, parameter-insensitive regularization using Dirichlet energy.
result The method preserves ordering and stability across different regularization strengths.
Geometric theory of projection heads in self-supervised learning.
problem Dimensional collapse and information invariance trade-off in projection heads.
method Geometric modeling of projection heads as Riemannian metrics, analyzing Hessian eigenvalues, and tracking optimization geometry.
result Smooth nonlinear heads induce negative curvature, preventing collapse; linear and ReLU heads cannot.
The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique. By using a neural inference network to approximate the model's posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be opti…
Study on deep multi-head self-attention dynamics, proving homogenized limits under specific scalings.
problem Understanding the behavior of deep multi-head self-attention models as depth increases.
method Random model of deep multi-head self-attention, viewing depth as time, and analyzing the residual stream as a particle system.
result Homogenized limit of the dynamics, leading to deterministic or stochastic behavior depending on scaling, with implications for representation collapse.
Improved RL training for DMs reduces mode collapse and preserves diversity.
problem Mode collapse and training instability in RL fine-tuned diffusion models.
method Dynamic hierarchical RL training with sliding-window parameter regularisation.
result Models trained with HRF achieve better preservation of diversity in downstream tasks.
We introduce a Bayesian Gaussian process latent variable model that explicitly captures spatial correlations in data using a parameterized spatial kernel and leveraging structure-exploiting algebra on the model covariance matrices for computational tractability. Inference is made tractable through a collapsed variation…
NS-GAN mode collapse due to sample weighting inversion, solved with MM-nsat.
problem Mode collapse in GANs due to sample weighting inversion.
method Preserves MM-GAN sample weighting while avoiding saturation by rescaling gradients.
result MM-nsat improves mode coverage, stability, and FID on MNIST and CIFAR-10.
New insights explain why β-VAEs fail at disentanglement.
problem Disentanglement performance of β-VAEs peaks at intermediate β and collapses as regularization increases. method Formalized information-theoretic mechanism, introduced λβ-VAE to stabilize disentanglement. result Strong regularization pressure leads to mutual information collapse in β-VAEs. Vacuum gravity shows black holes can form without collapse.
problem Formation of black holes in vacuum gravity.
method Cauchy--double-null framework, Yau's boundary criterion.
result Black holes form from boundary effects in vacuum spacetime.
New law predicts first extinction in resampling processes.
problem Intractable extinction times in resampling processes.
method Modeling multinomial updates as independent square-root diffusions.
result Closed-form law for first-extinction time with linear cost.
We propose an efficient inference method for switching nonlinear dynamical systems. The key idea is to learn an inference network which can be used as a proposal distribution for the continuous latent variables, while performing exact marginalization of the discrete latent variables. This allows us to use the reparamet…
SGD tends to favor simpler subnetworks, improving generalization.
problem SGD's tendency to favor simpler subnetworks over complex ones.
method Identifying invariant sets and analyzing SGD's behavior around them.
result SGD collapses networks to simpler subnetworks, improving generalization.
Data analysis with log-periodical parametrization of the Brent oil price dynamics has allowed to estimate (very approximately) the date when the dashing collapse of the Brent oil price will achieve the absolute minimum level (corresponding to the so-called singularity point), after which there will occur a rather rapid…
The paper shows how training with synthetic data can lead to model improvement, not degradation, under certain conditions.
problem Model collapse in iterative training on contaminated sources.
method Statistical analysis of iterative training on a mixture of true and synthetic data.
result Training with synthetic data can lead to model improvement, not degradation, under specific conditions.
Bayesian deep learning faces posterior collapse due to likelihood vs. prior competition.
problem Posterior collapse in Bayesian deep learning models.
method Identified competition between likelihood and prior regularization in a linear latent variable model.
result Posterior collapse is related to neural and dimensional collapse, suggesting a broader learning issue.
Large scale networks delineating collective dynamics often exhibit cascading failures across nodes leading to a system-wide collapse. Prominent examples of such phenomena would include collapse on financial and economic networks. Intertwined nature of the dynamics of nodes in such network makes it difficult to disentan…
Jeffreys Flow improves robustness of Boltzmann generators for rare event sampling.
problem Rare events and metastable trapping in sampling physical systems with rough energy landscapes.
method Introduces Jeffreys Flow, a robust generative framework using Parallel Tempering distillation and symmetric Jeffreys divergence to mitigate mode collapse and improve mode coverage.
result Minimizing Jeffreys divergence suppresses mode collapse and corrects inaccuracies in multi-modal distributions.
Proves weakly non-collapsed RCD spaces are strongly non-collapsed.
problem Proving the equivalence of weakly non-collapsed and strongly non-collapsed RCD spaces.
method Analyzes properties of RCD spaces and uses auxiliary results.
result Confirms conjecture about RCD spaces being strongly non-collapsed.
We study the dynamics of exchange value in a system composed of many interacting agents. The simple model we propose exhibits cooperative emergence and collapse of global value for individual goods. We demonstrate that the demand that drives the value exhibits non Gaussian "fat tails" and typical fluctuations which gro…
Study on Neural Collapse limits in deep learning.
problem Understanding the limits of Neural Collapse in deep learning.
method Investigated Neural Collapse in the context of generalization and feature learning, refining conjectures and conducting experiments.
result Neural Collapse primarily occurs on the train set and not on the test set, suggesting it is an optimization phenomenon with unclear connections to generalization.
Neural networks parameterize time-varying Markov dynamics in financial time series.
problem Estimating Markov transition matrices in high-resolution, high-noise financial data.
method Introduces a neural network framework to generate explicit, time-varying Markov transition matrices, constraining neural outputs to formal stochastic operators.
result Learned operators capture regime shifts, with high-volatility regimes homogenizing transition dynamics.
Study models deep learning training dynamics using locally elastic SDEs to reveal feature separability.
problem Understanding how deep learning models separate features from different classes during training.
method Modeling deep learning training using locally elastic SDEs with a drift term reflecting backpropagation impact.
result Local elasticity in SDEs leads to linear separability of features, resulting in vanishing training loss.
Ancient solutions found on flag manifolds from invariant Einstein metrics.
problem Understanding the behavior of Ricci flow on flag manifolds.
method Global study of the dynamical system induced by the Ricci flow, using invariant Einstein metrics and Poincaré compactification.
result Non-collapsed ancient solutions emerge from invariant Einstein metrics, with a Type I singularity in finite time.
Study shows formation of Kerr black holes with complete apparent horizons and proves Penrose inequalities.
problem Formation of Kerr black holes and Penrose inequalities.
method Combining gravitational-collapse and Kerr stability results with new coordinate changes and elliptic arguments.
result Proves dynamical and spacetime Penrose inequalities in black hole formation spacetimes.
We investigate a multi-household DSGE model in which past aggregate consumption impacts the confidence, and therefore consumption propensity, of individual households. We find that such a minimal setup is extremely rich, and leads to a variety of realistic output dynamics: high output with no crises; high output with i…