The paper explains how continuous language models can produce discrete, interpretable meanings.
problem Semantic collapse in continuous systems of large language models.
method Formalizing large language models as Continuous State Machines (CSMs) and analyzing the associated transfer operator.
result The leading eigenfunctions of the transfer operator induce a finite number of invariant meaning basins, explaining how continuous computation can produce discrete, interpretable semantics.
Variational autoencoders learn unsupervised data representations, but these models frequently converge to minima that fail to preserve meaningful semantic information. For example, variational autoencoders with autoregressive decoders often collapse into autodecoders, where they learn to ignore the encoder input. In th…
A simple pooling technique prevents posterior collapse in sequence VAEs.
problem Posterior collapse in sequence VAEs, causing models to ignore latent variables.
method Proposed a pooling technique to prevent posterior collapse.
result Significantly better data log-likelihood achieved compared to standard sequence VAEs.
Introduces PCG for better counterfactual explanations in vision models.
problem Ambiguity in latent-space optimization methods for counterfactual explanations.
method Constructs counterfactuals by tracing geodesics under a perceptually Riemannian metric.
result PCG outperforms baselines and reveals hidden failure modes.
New findings suggest non-contrastive learning has many bad minima, not just collapsed ones.
problem The effectiveness of non-contrastive learning in unsupervised feature learning.
method Theoretical analysis and controlled experiments on simple data models.
result Non-contrastive losses have a preponderance of non-collapsed bad minima, and these minima are not avoided during training.
Geometric theory of projection heads in self-supervised learning.
problem Dimensional collapse and information invariance trade-off in projection heads.
method Geometric modeling of projection heads as Riemannian metrics, analyzing Hessian eigenvalues, and tracking optimization geometry.
result Smooth nonlinear heads induce negative curvature, preventing collapse; linear and ReLU heads cannot.
New insights explain why β-VAEs fail at disentanglement.
problem Disentanglement performance of β-VAEs peaks at intermediate β and collapses as regularization increases. method Formalized information-theoretic mechanism, introduced λβ-VAE to stabilize disentanglement. result Strong regularization pressure leads to mutual information collapse in β-VAEs. CADD improves generative quality by augmenting discrete diffusion with continuous latent space.
problem Loss of semantic information between denoising steps in discrete diffusion models.
method Introduces a framework that augments discrete state space with a continuous latent space, allowing for graded, informative masked tokens.
result CADD improves generative quality across text generation, image synthesis, and code modeling.
Intelligence emerges from stabilizing invariant cycles in memory.
problem Understanding the nature of intelligence and its emergence.
method Structural-dynamical account rooted in a topological closure law: \(\partial^2=0\).
result Memory-amortized inference (MAI) mechanism that implements SbS \(
ightarrow\) CCUP.
DCAE learns compact latent representations for one-class novelty detection.
problem Learning compact latent representations for one-class novelty detection.
method DCAE learns compact and collapse-free latent representations through internal discriminative layers of GANs, reconstructing in-class data finely and exclusively.
result DCAE achieves state-of-the-art performance on novelty and adversarial example detection.
Proposes a new model for clustering passenger trajectories with graphs.
problem Hierarchical trip structure, inaccurate clustering number, and lack of spatial semantic graphs.
method Tensor Dirichlet Process Multinomial Mixture model with graphs and a tensor version of Collapsed Gibbs Sampling.
result Automatic determination of the number of clusters and better cluster quality.
Probabilistic topic models are popular unsupervised learning methods, including probabilistic latent semantic indexing (pLSI) and latent Dirichlet allocation (LDA). By now, their training is implemented on general purpose computers (GPCs), which are flexible in programming but energy-consuming. Towards low-energy imple…
LLMs outperform strong tabular baselines on industrial car retrofit prediction.
problem Industrial retrofit planning on structured operational data
method Embedding features, direct prompted classification, and ML+LLM stacking
result LLMs outperform strong tabular baselines on industrial car retrofit prediction
The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-te…
Latent Dirichlet Allocation (LDA) is a popular topic modeling technique for discovery of hidden semantic architecture of text datasets, and plays a fundamental role in many machine learning applications. However, like many other machine learning algorithms, the process of training a LDA model may leak the sensitive inf…
Bayesian deep learning faces posterior collapse due to likelihood vs. prior competition.
problem Posterior collapse in Bayesian deep learning models.
method Identified competition between likelihood and prior regularization in a linear latent variable model.
result Posterior collapse is related to neural and dimensional collapse, suggesting a broader learning issue.
Semantic TrueLearn uses semantic graphs to improve educational recommendation systems.
problem Challenges in handling semantic and hierarchical structure in knowledge areas.
method Introduces a novel learner model that exploits semantic relatedness between knowledge components using a Wikipedia link graph.
result Achieves statistically significant improvements in predictive performance for educational engagement.
Proves weakly non-collapsed RCD spaces are strongly non-collapsed.
problem Proving the equivalence of weakly non-collapsed and strongly non-collapsed RCD spaces.
method Analyzes properties of RCD spaces and uses auxiliary results.
result Confirms conjecture about RCD spaces being strongly non-collapsed.
Study on Neural Collapse limits in deep learning.
problem Understanding the limits of Neural Collapse in deep learning.
method Investigated Neural Collapse in the context of generalization and feature learning, refining conjectures and conducting experiments.
result Neural Collapse primarily occurs on the train set and not on the test set, suggesting it is an optimization phenomenon with unclear connections to generalization.
IdBench benchmarks semantic representations of identifiers, revealing strengths and weaknesses.
problem Evaluating semantic representations of identifiers in source code.
method Created a benchmark using developer ratings, evaluated natural language and source code embeddings, and compared lexical string distance functions.
result No single technique provides a satisfactory representation of semantic similarities, but ensemble models can improve performance.
Ricci flow smooths locally collapsing manifolds with controlled curvature.
problem Locally collapsing manifolds with controlled Ricci curvature.
method Ricci flow for a definite period of time, detecting collapsing infranil fiber bundles.
result Topological conditions detect collapsing infranil fiber bundles.
Optimal Transport Graph Neural Networks (OT-GNN) improves graph embeddings by using optimal transport.
problem Graph Neural Networks (GNN) often lose structural or semantic information when aggregating node embeddings.
method Combines optimal transport (OT) with parametric graph models to compute graph embeddings from Wasserstein distances between node embeddings and prototype point clouds.
result OT-GNN outperforms popular methods on molecular property prediction tasks and produces smoother graph representations.
Mathematical analysis shows annealing prevents mode collapse in Gaussian mixtures.
problem Mode collapse in variational inference for multimodal distributions.
method Analyzed annealing strategies for Gaussian mixtures, derived formulas, and tested on neural networks.
result Appropriately chosen annealing schemes can robustly prevent mode collapse.
The study characterizes and rules out collapsing in convex ancient mean curvature flow.
problem Characterizing and ruling out collapsing in convex ancient mean curvature flow.
method Characterization and counterexamples.
result Collapsing occurs if and only if the flow is asymptotic to at least one Grim hyperplane.
Despite excellent progress in recent years, mode collapse remains a major unsolved problem in generative adversarial networks (GANs).In this paper, we present spectral regularization for GANs (SR-GANs), a new and robust method for combating the mode collapse problem in GANs. Theoretical analysis shows that the optimal …
New method controls posterior collapse in VAEs without network architecture constraints.
problem Posterior collapse in VAEs reduces diversity of generated samples.
method Introduces Latent Reconstruction (LR) loss to control posterior collapse.
result Controls posterior collapse on various datasets without architectural constraints.
Transfer learning aims at building robust prediction models by transferring knowledge gained from one problem to another. In the semantic Web, learning tasks are enhanced with semantic representations. We exploit their semantics to augment transfer learning by dealing with when to transfer with semantic measurements an…
Study flat manifolds' collapsed limits as flat orbifolds.
problem Understanding collapsed limits of flat manifolds.
method Analyzing totally geodesic foliations and Gromov-Hausdorff limits.
result Identify collapsed limits as flat orbifolds and provide criteria for singularity.
Special Lagrangian submanifolds emerge from K3 surface collapse.
problem Understanding special Lagrangian submanifolds in K3 surface collapse.
method Lifting affine lines to degenerating sequences of special Lagrangian submanifolds.
result Constructing special Lagrangian two-spheres connecting Taub-NUT bubbles.
The torus cannot collapse to a segment under certain curvature conditions.
problem Impossibility of codimension-one collapse for surfaces of negative Euler characteristic.
method Analysis of Gaussian curvature and homology loops.
result The torus cannot collapse to a segment under similar conditions to surfaces of negative Euler characteristic.
Collapsibility is a combinatorial strengthening of contractibility. We relate this property to metric geometry by proving the collapsibility of any complex that is CAT(0) with a metric for which all vertex stars are convex. This strengthens and generalizes a result by Crowley. Further consequences of our work are: (1) …
Two-dimensional collapsed spaces with lower Ricci bounds are topological surfaces.
problem Topology of collapsed spaces with lower Ricci bounds
method Prove that collapsed spaces are topological surfaces
result Collapsed spaces are topological surfaces
This paper presents a Semantic Attribute Modulation (SAM) for language modeling and style variation. The semantic attribute modulation includes various document attributes, such as titles, authors, and document categories. We consider two types of attributes, (title attributes and category attributes), and a flexible a…
Prove that collapsing CSC metrics can be perturbed to invariant collapsing CSC metrics.
problem Prove that collapsing constant scalar curvature metrics can be perturbed to invariant collapsing constant scalar curvature metrics.
method Prove that a sequence of constant scalar curvature metrics which is collapsing with bounded curvature to a manifold can be perturbed to a sequence of invariant collapsing constant scalar curvature metrics.
result Prove that a sequence of constant scalar curvature metrics which is collapsing with bounded curvature to a manifold can be perturbed to a sequence of invariant collapsing constant scalar curvature metrics.
The paper shows how integrating categorical semantics can enhance unsupervised domain translation.
problem Improving unsupervised domain translation between perceptually different domains.
method Learning invariant categorical semantic features in an unsupervised manner and conditioning them on the style encoder.
result Conditioning the style encoder on learned categorical semantics improves translation and stylization.
We will simplify the earlier proofs of Perelman's collapsing theorem of 3-manifolds given by Shioya-Yamaguchi and Morgan-Tian. Among other things, we use Perelman's semi-convex analysis of distance functions to construct the desired local Seifert fibration structure on collapsed 3-manifolds. The verification of Perelma…
Estimate collapsibility of causal effects in CPDAGs via strong d-convex hulls.
problem Estimate causal effects in CPDAGs.
method Use strong d-convex hulls to characterize minimal collapsible sets.
result Efficient algorithm for obtaining collapsible sets in DAGs and CPDAGs.
We introduce the theory of strong homotopy types of simplicial complexes. Similarly to classical simple homotopy theory, the strong homotopy types can be described by elementary moves. An elementary move in this setting is called a strong collapse and it is a particular kind of simplicial collapse. The advantage of usi…
Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
problem Understanding and optimizing deep learning training phases.
method Direct measurements on three deepnet architectures across seven datasets.
result Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
Lower Ricci curvature bound prevents first Betti number from dropping more than dimension in collapsing manifolds.
problem Understanding how the first Betti number behaves under manifold collapse with Ricci curvature bounds.
method Analyzing sequences of Riemannian manifolds with lower Ricci curvature bounds.
result The first Betti number cannot drop more than the dimension in collapsing manifolds.
Study tackles criterion collapse in learning criteria, showing conditions for loss minimization.
problem Criterion collapse in optimization, focusing on error probability minimizers.
method Analyzes various learning criteria, including DRO, OCE risks, and non-monotonic criteria.
result Non-monotonic criteria can avoid collapse, while monotonic ones cannot.
Study on collapsing Calabi-Yau manifolds and their metrics.
problem Understanding degenerations of Calabi-Yau manifolds with Ricci-flat Kahler metrics.
method Survey of recent developments, focusing on volume collapsing metrics.
result New insights into the behavior of Calabi-Yau manifolds under volume collapse.
This work provides uncertainty intervals for semantic latent variables in disentangled latent spaces.
problem Challenges in providing meaningful uncertainty quantification for semantic information in disentangled latent spaces.
method Uses quantile regression to output heuristic uncertainty intervals, calibrates these intervals to contain true latent values, and propagates them through the generator.
result Reliably communicates semantically meaningful, principled, and instance-adaptive uncertainty in image super-resolution and image completion.
Semantify-NN verifies neural network robustness against semantic perturbations.
problem Verifying robustness of neural networks against semantic adversarial attacks.
method Inserting semantic perturbation layers (SP-layers) into neural networks to verify robustness.
result Semantify-NN significantly improves robustness verification performance over ℓp-norm-based methods. This is an expositiry article on collapsing theory in Riemannian geometry written for the Modern Encyclopedia of Mathematical Physics (MEMPhys). We focus on describing the geometric and topological structure of collapsed/non-collapsed regions in Riemannian manifold under various curvature assumptions. Numerous applicat…
In this paper we extend the works of Tancer and of Malgouyres and Francés, showing that (d,k)-collapsibility is NP-complete for d≥k+2 except (2,0). By (d,k)-collapsibility we mean the following problem: determine whether a given d-dimensional simplicial complex can be collapsed to some k-dimensional sub…
New system preserves message meaning in wireless networks, improving data rate.
problem Efficiently transmitting message meaning in wireless networks.
method Modeling semantics as hidden random variables, using Information Bottleneck for compression.
result 20 dB SNR improvement for semantic communication.
New diffusion models improve counterfactual image generation with semantic control.
problem Challenges in preserving identity, maintaining quality, and ensuring causal model faithfulness in counterfactual image generation.
method Integrates semantic representations into diffusion models through Pearlian causality, introducing spatial, semantic, and dynamic abduction.
result Demonstrates high-level semantic identity preservation and principled trade-offs between faithful causal control and identity preservation.