The paper applies math and physics to language models, introducing entropy and geometric concepts.
problem Understanding and improving language models to approximate intelligent language.
method Formal definitions, functional analysis, topology, thermodynamics, and set theory.
result Entropy function reveals key obstacles for LLMs and offers insights into language models.
Introduces Kundt spaces using geometric language.
problem Clarify properties of Kundt spaces.
method Geometric approach focusing on key concepts.
result Pedagogical introduction to Kundt spaces.
We propose a novel geometric approach for learning bilingual mappings given monolingual embeddings and a bilingual dictionary. Our approach decouples learning the transformation from the source language to the target language into (a) learning rotations for language-specific embeddings to align them to a common space, …
Study explains Zipf's law using geometric mechanisms from a finite alphabet.
problem Explains Zipf's law in language without relying on linguistic elements.
method Uses the Full Combinatorial Word Model (FCWM) to generate geometric distributions of word lengths.
result Supports predictions of power-law rank-frequency curves, matching various languages.
I introduce a new geometrical approach to thermo--statistical mechanics. Here I highlight the main physical ideas, and how do they translate into geometrical language. I contrast the present approach with previous thermo--statistical--geometrical formalisms, (pseudo-)Riemannian [Weinhold 1975; Ruppeiner 1979] as well a…
Geometric stability predicts steerability and detects drift in language models.
problem Predicting steerability and detecting drift in language models.
method Supervised and unsupervised geometric stability measures.
result Supervised geometric stability predicts steerability with high accuracy and detects drift earlier.
Geometric approach for unsupervised word embedding alignment.
problem Learning alignment between word embeddings of source and target languages.
method Formulates alignment as domain adaptation on the manifold of doubly stochastic matrices, employing Riemannian conjugate gradient algorithm.
result Empirically outperforms state-of-the-art methods on bilingual lexicon induction tasks.
Study describes singularities of height functions on specific singular surfaces.
problem Analyzing singularities of height functions on singular surfaces.
method Using geometric language and blowing-ups, investigate singularities of height functions and dual surfaces.
result Characterized singularities of height functions and dual surfaces on specific singular surfaces.
Let Γ be the fundamental group of a manifold modeled on three dimensional Sol geometry. We prove that Γ has a finite index subgroup G which has a rational growth series with respect to a natural generating set. We do this by enumerating G by a regular language. However, in contrast to most earlier proofs of thi…
This paper addresses anisotropy in Transformer models, providing geometric insights and empirical support.
problem Anisotropy phenomenon in Transformer models, challenging their geometric interpretation.
method Derive geometric arguments and use concept-based mechanistic interpretability during training.
result Activation-derived directions capture large gradient energy and a larger share of gradient anisotropy than normal controls.
This note, in a rather expository manner, serves as a conceptional introduction to the certain underlying mathematical structures encoding the geometric quantization formalism and the construction of Witten's quantum invariants, which is in fact organized in the language topological quantum field theory.
The paper explores linear representations in language models using counterfactuals.
problem Understanding linear representations and geometric concepts in large language models.
method Formalized linear representation in output and input spaces, identified causal inner product.
result Unified understanding of linear representations and their connection to interpretation and control.
Stan is a probabilistic programming language that is popular in the statistics community, with a high-level syntax for expressing probabilistic models. Stan differs by nature from generative probabilistic programming languages like Church, Anglican, or Pyro. This paper presents a comprehensive compilation scheme to com…
Discrete Flow Maps bypass sequential prediction limits for parallel text generation.
problem Sequential autoregressive prediction limits large language model speed.
method Flow Maps compress generative trajectories into single-step mappings.
result Discrete Flow Maps surpass previous state-of-the-art results in discrete flow modeling.
Researchers analyze the structure of token spaces in large language models.
problem Understanding the geometric and topological structure of token spaces in large language models.
method Developed estimators for the dimension and Ricci scalar curvature of the token subspace and applied them to three large language models.
result The token subspace is not a manifold but a stratified manifold with significantly negative Ricci curvature.
This paper examines to what degree current deep learning architectures for image caption generation capture spatial language. On the basis of the evaluation of examples of generated captions from the literature we argue that systems capture what objects are in the image data but not where these objects are located: the…
Introduces optimization geometrodynamics for dynamic geometric optimization.
problem Gradient-based optimization methods struggle with changing geometric constraints.
method Optimization geometrodynamics separates invariant and improvable geometric mismatches.
result Dynamic geometric complexity measures the minimum geometric cost to reduce optimization difficulty.
ATD measures language distance using neural models, recovering linguistic groupings.
problem Lack of a unified quantitative measure for cross-linguistic distance.
method Pretrained multilingual language models, attention mechanisms, optimal transport.
result ATD quantifies representational distance between languages, recovering linguistic groupings.
Geo2DR learns graph representations using substructure patterns.
problem Learning distributed representations of graphs efficiently.
method Unsupervised learning with discrete substructure patterns and neural language models.
result Geo2DR achieves high reproducibility and interoperability in graph classification.
Proposes integrating global and local entropy for more reliable LLMs.
problem Uncertainty in large language models (LLMs) leads to unreliable predictions.
method Measures global uncertainty from hidden-state matrices and local uncertainty from tokens, combining them via a multiplicative gate.
result Global-Local Uncertainty (GLU) outperforms unsupervised baselines across multiple models and benchmarks.
We propose a new statistical model suitable for machine learning of systems with long distance correlations such as natural languages. The model is based on directed acyclic graph decorated by multi-linear tensor maps in the vertices and vector spaces in the edges, called tensor network. Such tensor networks have been …
Study matches two noisy point clouds with geometric transformations and relabeling.
problem Matching two noisy point clouds with orthogonal transformations and relabeling.
method Information-theoretic results and Ping-Pong algorithm for computational alignment.
result The Ping-Pong algorithm retrieves the planted signal after one step.
We employ the language of Cartan's geometry to present a model for studying vector spaces of Killing two-tensors defined in pseudo-Riemannian spaces of constant curvature under the action of the corresponding isometry group. We also discuss geometric properties of joint invariants of Killing two-tensors defined in the …
Hyperfitting improves LLM generation quality by enhancing diversity, contrary to simple temperature scaling.
problem Improving open-ended generation quality of LLMs with minimal fine-tuning effort.
method Demonstrates that hyperfitting, a phenomenon where LLMs are fine-tuned to near-zero training loss, enhances generation quality and mitigates repetition.
result Hyperfitting is distinct from temperature scaling and involves a dynamic, context-dependent rank reordering mechanism in the final transformer block.
An important result from psycholinguistics (Griffiths & Kalish, 2005) states that no language can be learned iteratively by rational agents in a self-sustaining manner. We show how to modify the learning process slightly in order to achieve self-sustainability. Our work is in two parts. First, we characterize iterated …
A Python tool generates synthetic data for cluster analysis from high-level descriptions.
problem Creating synthetic data for cluster analysis is laborious and requires detailed geometric parameters.
method Proposes natural language-based synthetic data generation and implements it in a Python package.
result Makes it easy to set up interpretable and reproducible benchmarks for cluster analysis.
Direct Feedback Alignment performs well on diverse deep learning tasks and architectures.
problem The limitations of backpropagation in parallelizing and scaling to modern deep learning tasks.
method Direct Feedback Alignment approach applied to neural view synthesis, recommender systems, geometric learning, and natural language processing.
result Direct Feedback Alignment successfully trains a wide range of state-of-the-art deep learning architectures with performance close to fine-tuned backpropagation.
Researchers use LLMs to judge other LLMs, but this study provides a new geometric perspective to understand when it works.
problem The challenge of evaluating LLMs using other LLMs as judges, considering both aleatoric and epistemic uncertainties.
method A geometric perspective on ranking LLM candidates using probability simplices, analyzing conditions for identifiable rankings and designing Bayesian priors.
result Geometric analysis reveals that rankings based on LLM judges are robust in many but not all datasets, emphasizing the importance of modeling epistemic uncertainty.
Newtonian, Lagrangian, and Hamiltonian dynamical systems are well formalized mathematically. They give rise to geometric structures describing motion of a point in smooth manifolds. Riemannian metric is a different geometric structure formalizing concepts of length and angle. The interplay of Riemannian metric and its …
Einstein gravity in both 3 and 4 dimensions, as well as some interesting generalizations, can be written as gauge theories in which the connection is a Cartan connection for geometry modeled on a symmetric space. The relevant models in 3 dimensions include Einstein gravity in Chern-Simons form, as well as a new formula…
In differential-geometric language, vortex-lines equations on extended phase space of a system may be written as iγ˙dσ=0, where σ is a differential 1-form. This is the structure, to give a paradigmatic example, of the Hamilton equations. Here, we study equations of the same structure, where σ is a differen…
Quaternionic differential geometry expands geometric concepts using quaternions.
problem Generalizing geometric concepts to quaternionic constraints.
method Generalizing curves and surfaces, curvature, torsion, differential forms, and directional derivatives to quaternionic constraints.
result Quaternionic formalism provides a suitable language for differential geometry.
The paper uncovers symmetries in large language models through layer-peeled optimization.
problem Understanding geometric structure in large language model weights and context embeddings.
method Constrained layer-peeled optimization program to analyze symmetries in next-token distributions.
result Symmetries in target next-token distributions are transferred to optimal model weights and context embeddings.
Study on geometric variational problems for existence, regularity, and uniqueness of solutions.
problem Geometric variational problems, focusing on existence, regularity, and uniqueness of solutions.
method Formulated in Federer and Fleming's theory of currents, discussed the existence theory, and presented core ideas of the (interior) regularity theory for area-minimizing currents and optimal transport paths. Two original results on generic uniqueness of solutions were presented.
result Generic uniqueness of solutions for both Plateau's problem and optimal branched transport problem.
Transformer model for probabilistic dynamical systems.
problem Modeling high-dimensional dynamical systems from noisy observations.
method Parallel between dynamical systems and language modeling; transformer-based model with geometrical properties; iterative training algorithm.
result Fine-grid approximation of conditional probabilities for high-dimensional systems.
Randomized Geometric Algebra for Convex Neural Networks Optimizes Transfer Learning.
problem Training neural networks to global optimality via convex optimization.
method Randomized algorithms in Clifford's Geometric Algebra for hypercomplex vector spaces.
result Convex optimization and geometric algebra improve LLMs' robustness and reliability in transfer learning.
Introduces a new geometric framework for non-perturbative BV-theory.
problem Non-perturbative generalization of BV-theory in infinite-dimensional spaces.
method Derived differential geometry and homotopical algebraic geometry.
result Concrete model of derived smooth stacks for encoding non-perturbative BV-theory.
GeLoRA optimizes LoRA fine-tuning by dynamically adjusting ranks based on intrinsic dimensionality.
problem Efficient fine-tuning of large language models with limited computational resources.
method GeLoRA computes intrinsic dimensionality to adaptively select LoRA ranks, balancing expressivity and efficiency.
result GeLoRA consistently outperforms recent baselines within the same parameter budget on multiple tasks.
Geometric Brownian motion simulates stock prices for Brazilian small caps index.
problem Simulating stock prices for the Brazilian small caps index.
method Used geometric Brownian motion to simulate stock prices of Brazilian small caps index using historical data.
result Simulated prices better for portfolios with higher returns, lower risks, and higher Sharpe Indexes.
Motion planning and control are key problems in a collection of robotic applications including the design of autonomous agile vehicles and of minimalist manipulators. These problems can be accurately formalized within the language of affine connections and of geometric control theory. In this paper we overview recent r…
This is a survey on the geometry of warped products, without, or essentially with only soft, calculation. Somewhere in the paper, the goal was to give a synthetic account since existing approaches are rather analytic. Somewhere else, we have interpreted statements, especially by means of a physical terminology. This is…
Applying the classical Serre-Swan theorem, as this is extended to topological (non-normed) algebras, one attains a classification of elementary particles via their spin-structure. In this context, our argument is virtually based on a ``correspondence principle'' of S. A. Selesnick, formulated herewith in a sheaf-theore…
Geometric arbitrage theory reformulates a generic asset model possibly allowing for arbitrage by packaging all asset and their forward dynamics into a stochastic principal fibre bundle, with a connection whose parallel transport encodes discounting and portfolio rebalancing, and whose curvature measures, in this geomet…
Deep neural networks favor symmetric structures, enabling multilevel symmetries.
problem Understanding and optimizing deep neural networks.
method Formulating DNN training as convex Lasso problems with geometric algebra.
result Deep networks inherently favor symmetric structures, enabling multilevel symmetries.
Graphs help agents learn emergent communication.
problem Learning robust communication protocols in multi-agent systems.
method Graph convolutional networks to model emergent language and cooperation.
result Agents learn to generalize beyond training samples, revealing true factors of variation.
This paper connects geometric diagrams to spherical T-duality.
problem Establishing equivalence between geometric and string-theory frameworks.
method Introducing ⋆-diagrams and proving their equivalence to spherical T-dual pairs. result Concrete geometric realization of spherical T-duality.
Survey on geometric foundations of data reduction methods.
problem High-dimensional data with intrinsic nonlinear structure.
method Spectral manifold learning methods.
result Derivation and convergence analysis of spectral manifold learning.
The associator of a non-associative algebra is the curvature of the Hochschild quasi-complex. The relationship ``curvature-associator'' is investigated. Based on this generic example, we extend the geometric language of vector fields to a purely algebraic setting, similar to the context of Gerstenhaber algebras. We int…