Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
We show that the quantum field theoretical formulation of the τ-function theory has a geometrical interpretation within the classical transformation theory of conjugate nets. In particular, we prove that i) the partial charge transformations preserving the neutral sector are Laplace transformations, ii) the basic ver…
Geometric approach uses Bäcklund transformations to create integrable discrete analogs of surface nets.
problem Creating integrable discrete analogs of surface nets and conjugate nets.
method Interpreting classical differential geometry results through Bäcklund transformations and applying permutability properties.
result Integrable discrete analogs of asymptotic and conjugate nets are constructed.
A Darboux transformation for polarized space curves is introduced and its properties are studied, in particular, Bianchi permutability. Semi-discrete isothermic surfaces are described as sequences of Darboux transforms of polarized curves in the conformal n-sphere and their transformation theory is studied. Semi-discre…
Kendall transformation converts continuous data into categorical vectors for robust information theory.
problem Handling small number of observations and preserving ranking in continuous data.
method Kendall transformation converts ordered features into categorical vectors of pairwise order relations.
result Kendall transformation makes information theory methods applicable to continuous data robustly.
Extends cohomology theory for infinite volume transformation groups.
problem Generalizing cohomology for infinite volume transformation groups.
method Introduces norm-controlled cohomology as a generalization of bounded cohomology.
result Establishes norm-controlled cohomology for infinite volume transformation groups.
New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
The study explains transformer scaling laws using statistical and approximation theories.
problem Understanding why transformer scaling laws exist for large models trained on low-dimensional data.
method Established statistical estimation and mathematical approximation theories for transformers on low-dimensional manifolds.
result Predicted a power law between generalization error and model and data sizes, with power depending on intrinsic data dimension.
Unified description of Weierstrass-type representations.
problem Creating surfaces with special curvature properties.
method Unified description using classical transformation theory of Ω-surfaces.
result Unified understanding of Weierstrass-type representations.
Gauge theory explains Lie applicable surfaces.
problem Understanding Lie applicable surfaces.
method Gauge-theoretic approach.
result Coincides with classical notions of Ω- and Ω0-surfaces. Transformers learn to cluster Gaussian mixtures as well as the EM algorithm.
problem Learning guarantees of Transformers in multi-class clustering of Gaussian mixtures.
method Developed a theory connecting Transformer's Softmax Attention layers to the EM algorithm's workflow.
result Transformers achieve minimax optimal rate for clustering Gaussian mixtures with sufficient training samples and initialization.
In the double field theory, gauge symmetries are realized as generalized diffeomorphisms in the doubled spacetime. By consistency of the theory, dependence of tensor fields on the doubled coordinates is strongly constrained. This causes finite transformation law highly complicated, both technically and conceptually. In…
Study examines how Lorentz transformations affect foliations in spacetime.
problem Understanding foliations and their relation to observers under Lorentz transformations.
method Exploration of kinematic characteristics of timelike congruences (observers) and their relation to foliations.
result Local Lorentz transformations impact the existence of transversal foliations.
Transformers predict scattering amplitudes in theoretical physics.
problem Computing exact coefficients of scattering amplitudes in N = 4 SYM theory.
method Applied Transformers to predict integer coefficients of scattering amplitudes.
result Transformers achieve high (> 98%) accuracy on predicting scattering amplitudes.
Survey on broken ray transforms and open problems.
problem Understanding broken ray transforms and their applications.
method Survey of recent developments and open problems.
result Discussion of open problems in broken ray transforms.
New equations define frustrated conformal transformations generating maps on ambitwistor space.
problem Defining new conformal transformations in twistor theory.
method Introducing frustrated conformal transformations involving instantons.
result Solutions to the new equation generate maps on ambitwistor space.
Study parametrized cobordism categories for smooth bundles, proving index theorem.
problem Characterize bivariant transformations in cobordism categories.
method Define parametrized cobordism categories and use bivariant theories.
result Prove Dwyer-Weiss-Williams index theorem for parametrized A-theory.
Study S-duality and supersymmetry on curved manifolds using localization.
problem Understanding S-duality and supersymmetry on curved manifolds.
method Localization and Fourier transform interpretation of S-duality.
result Evidence for interpreting S-duality as Fourier transform.
Diffusion Transformer captures spatial-temporal dependencies in sequential data.
problem Capturing rich spatial and temporal dependencies in sequential data.
method Established theoretical guarantees for diffusion transformers learning Gaussian process data.
result Spatial-temporal dependencies are captured within attention layers of diffusion transformers.
In many Lagrangian field theories one has a Poisson bracket defined on the space of local functionals. We find necessary and sufficient conditions for a transformation on the space of local functionals to be canonical in three different cases. These three cases depend on the specific dimensions of the vector bundle of …
We construct manifestly superconformal field theories in six dimensions which contain a non-Abelian tensor multiplet. In particular, we show how principal 3-bundles over a suitable twistor space encode solutions to these self-dual tensor field theories via a Penrose-Ward transform. The resulting higher or categorified …
Operational theory for 2-bundles defines connections and gauge transformations.
problem Defining connections and gauge transformations for 2-bundles.
method Operational framework based on derived Lie group.
result Original formulation of 2-connections and gauge transformations.
Formalizes learning algorithm invariances using category theory.
problem Understanding and characterizing invariances in learning algorithms.
method Using category theory to define and formalize invariances of learning algorithms.
result Illustrated and contrasted the invariances of linear regression and ridge regression.
Introduces pseudo-codecomposition of transformation groups.
problem Understanding and categorizing transformation groups.
method Introduces pseudo-codecomposition and analyzes properties of transformation groups.
result The class of pseudo-codecomposable transformation groups is a proper intermediate class.
GridPyM handles grid diagrams for knot theory.
problem Handling grid diagrams for knot theory.
method Generates and simplifies grids, models local transformations.
result Models local transformations between grid diagrams.
Transformers can adaptively select and perform various machine learning tasks in context.
problem Transformers' ability to perform multiple machine learning tasks without explicit prompting.
method Statistical theory and efficient implementation of in-context gradient descent, enabling transformers to implement various algorithms and tasks.
result Transformers can implement a broad class of standard machine learning algorithms in context, including algorithm selection.
New Hermite approximations accelerate convergence with adaptive coordinate transformations.
problem Accelerating convergence of spectral approximations for Hermite expansions.
method Using normalizing flows for adaptive coordinate transformations and deriving error estimates.
result Error estimates for Hermite expansions under adaptive coordinate transformations.
The class of Schoenberg transformations, embedding Euclidean distances into higher dimensional Euclidean spaces, is presented, and derived from theorems on positive definite and conditionally negative definite matrices. Original results on the arc lengths, angles and curvature of the transformations are proposed, and v…
The paper develops a framework for abstracting causal models using category theory.
problem Difficulties in changing the variables used to describe a system, especially from fine-grained to coarse-grained.
method Introduces a category of interventional causal models and uses enriched category theory to prove compositionality properties.
result Compositionality of model transformations is established, with bounded errors for each step.
Unified theory explains two failure modes of deep transformers and provides initialisation guidelines.
problem Two failure modes (rank collapse and entropy collapse) of self-attention layers in deep transformers.
method Analytical theory of signal propagation through deep transformers, using the Random Energy Model analogy.
result Simple algorithm to compute trainability diagrams for correct initialisation hyper-parameters.
New Transformers maintain Lipschitz continuity for robustness.
problem Ensuring robustness in Transformers for safety-sensitive applications.
method Introducing gradient-descent-type in-context Transformers with explicit Euler steps of negative gradient flows.
result Universal approximation theorem for Lipschitz continuous Transformers.
Transformers enable in-context learning with guarantees for a wide range of tasks.
problem How to enable in-context learning with transformers for various tasks.
method Developed a universal approximation theory integrating Barron's function approximation with transformer capabilities.
result Transformers can approximate any target function with vanishingly small risk using a few in-context examples.
Paper studies Transformer learning theory for Euclidean and Riemannian domains.
problem Understanding and optimizing Transformer networks for regression tasks.
method Constructive approximation framework using softmax partition of unity and attention mechanism.
result Transformer can achieve uniform ε-approximation error with minimal parameters.
New PAC-Bayes bounds derived using Legendre transform and f-divergences.
problem Deriving PAC-Bayes bounds under various assumptions.
method Combining Legendre transform and Fenchel--Young inequality to derive change-of-measure inequalities.
result Extended PAC-Bayesian guarantees under tailored assumptions.
We develop some theory of double fibration transforms where the cycle space is a smooth manifold and apply it to complex projective space.
Develops new approach to symmetry in field theory using Lie groupoids.
problem Implementing symmetry in classical field theory.
method Adapting Lie groupoid/algebroid formalism to gauge theories.
result Adapts formalism to gauge theories and proves minimal coupling and Utiyama's theorem.
Hurwitz transformations are defined as specific automorphisms of a Cayley-Dickson algebra. These transformations generate quadratic and nonquadratic forms. We investigate here the Hurwitz transformations corresponding to Cayley-Dickson algebras of dimensions 2m = 2, 4 and 8. The Hurwitz transformations which lead to qu…
Line graph transformation aids graph isomorphism tests by excluding challenging graph properties.
problem Limited theoretical understanding of line graph transformation's impact on GNN models.
method Examined CFI and strongly regular graphs, showing line graph transformation helps WL tests distinguish these graphs.
result Line graph transformation aids WL tests in distinguishing challenging graph properties.
Study fermionic theories, their anomalies, and modular transformations.
problem Understanding fermionic theories and their anomalies.
method Use spin-cobordisms, surgeries, and invertible topological quantum field theories.
result Explicit combinatorial expressions for spin-bordism invariants.
The study reveals how attention paths in Transformers influence learning outcomes.
problem Understanding the theoretical basis of Transformers' performance.
method Developed a statistical mechanics theory for a simplified attention network.
result The predictor statistics are influenced by the combination of attention paths.
In this paper we analyze supergeometric locally covariant quantum field theories. We develop suitable categories SLoc of super-Cartan supermanifolds, which generalize Lorentz manifolds in ordinary quantum field theory, and show that, starting from a few representation theoretic and geometric data, one can construct a f…
We study S-dualities in analytically continued SL(2) Chern-Simons theory on a 3-manifold M. By realizing Chern-Simons theory via a compactification of a 6d five-brane theory on M, various objects and symmetries in Chern-Simons theory become related to objects and operations in dual 2d, 3d, and 4d theories. For example,…
This paper is devoted to the development and applications of some (new) basic concepts in Lie theory, both from `computational" and "observability" viewpoint. We specify set of all "G-equivariant" maps from a given Lie group G to the underlying manifold M, namely G-set, and also we introduce "conjugacy" in Lie group …
Density destructors simplify complex PDFs to maximize entropy, linking to information theory.
problem Complex multivariate PDFs are hard to analyze.
method Invertible transforms that progressively remove structure from PDFs.
result Density destructors can improve estimates of information theoretic quantities.
Study on neural scaling laws for solving linear systems in-context.
problem Theoretical guarantees for solving linear systems using a linear transformer architecture.
method Neural scaling laws and task diversity for in-domain and out-of-domain generalization.
result Novel notion of task diversity for necessary and sufficient condition of generalization under task shifts.
Holonomy-preserving transformations help recover Alexander polynomials from graph zeta functions.
problem Recovering Alexander polynomials from graph zeta functions.
method Introducing holonomy to preserve zeta functions of matrix-weighted graphs and extending to group elements and quandles.
result Holonomy-preserving transformations correspond to transformations of group presentations and preserve the twisted Alexander polynomial.
The asymptotic lattices and their transformations are studied within the line geometry approach. It is shown that the discrete asymptotic nets are represented by isotropic congruences in the Plucker quadric. On the basis of the Lelieuvre-type representation of asymptotic lattices and of the discrete analog of the Mouta…
A description of how a theory of gravity can be considered as a gauge theory (in the sense of Trautman) of the Poincare' group is given. As a result, it is shown that a gauge theory of this kind is consistent with the Equivalence Principle only if the Lagrangian and the constraints are preserved not only by the gauge t…