Study examines infinite limits of transformer dynamics, identifying key parameterizations.
problem Understanding the training dynamics of transformer models in the feature learning regime.
method Analysis of infinite scaling limits using dynamical mean field theory.
result Identified parameterizations that admit well-defined infinite width and depth limits.
Study infinite-depth limits of neural networks with fixed width.
problem Understanding the behavior of neural networks as depth increases with fixed width.
method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.
result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.
Infinite CNNs lose spatial correlations, but can be restored by correlated weights.
problem Infinite CNNs lose spatial correlations, which are crucial for their performance.
method Introduced correlated weights to restore spatial correlations in infinite CNNs.
result Optimal performance is achieved with a moderate level of weight correlation.
Geometrically infinite Kleinain groups have nonconical limit sets with the cardinality of the continuum. In this paper, we construct a geometrically infinite Fuchsian group such that the Hausdorff dimension of the nonconical limit set equals zero. For finitely generated, geometrically infinite Kleinian groups, we prove…
Wide neural networks can benefit from multi-task learning in their infinite-width limit.
problem The generalization behavior of wide neural networks in multi-task learning settings.
method Optimizing wide ReLU neural networks with L2-regularization promotes multi-task learning in the infinite-width limit.
result An exact quantitative characterization of multi-task learning in the infinite-width limit of wide ReLU neural networks.
We show that the limiting Khovanov chain complex of any infinite positive braid categorifies the Jones-Wenzl projector. This result extends Lev Rozansky's categorification of the Jones-Wenzl projectors using the limiting complex of infinite torus braids. We also show a similar result for the limiting Lipshitz-Sarkar-Kh…
New framework for understanding infinite-width neural networks.
problem Understanding the infinite-width limit behavior of neural networks.
method General framework to study limit behavior of neural models based on hyperparameter scaling.
result Derives scaling for existing mean-field and neural tangent kernel limits and introduces new dynamically stable limits.
Paper analyzes infinite-width attention layers using Tensor Programs.
problem Capturing the infinite-width limit of attention layers.
method Tensor Programs framework to rigorously identify the limit distribution.
result Derives exact form of infinite-width limit distribution without Gaussian approximations.
Bayesian neural networks approximate Student-t processes in the infinite-width limit.
problem Modeling uncertainty in neural networks with greater flexibility.
method Extending asymptotic properties of Gaussian processes to Student-t processes in the infinite-width limit of BNNs.
result Posterior BNNs converge to Student-t processes in the infinite-width limit.
Study transverse measures on infinite type hyperbolic surfaces.
problem Characterize the cone of transverse measures on infinite type hyperbolic surfaces.
method Use inverse limits and geodesic laminations to describe and construct cones of transverse measures.
result Explicit descriptions and bases of cones of transverse measures exist for many laminations.
We define a limiting slN Khovanov-Rozansky homology for semi-infinite positive multi-colored braids, and we show that this limiting homology categorifies a highest-weight projector for a large class of such braids. This effectively completes the extension of Cautis' similar result for infinite twist braid…
ResNets approximate log-Gaussian at initialization, improving network performance.
problem Understanding the initialization behavior of deep neural networks like ResNets.
method Analyzing ReLU ResNets in the infinite-depth-and-width limit, showing log-Gaussian behavior.
result ResNets at initialization exhibit hypoactivation and interlayer correlations, which are not captured by Gaussian limits.
Stable processes emerge as limits of deep neural networks with symmetric stable distributions.
problem Understanding the behavior of deep neural networks as they become infinitely wide.
method Analyzing fully connected feed-forward deep neural networks with symmetric stable distributions and showing the limit as a stable process.
result The infinite wide limit of the network is a stable process with multivariate stable distributions.
One of the goals of this article is to define a an unified setting adapted to the description of means (normalized integrals or invariant means) on an infinite product of measured spaces with infinite measure. We first remark that some known examples coming from the theory of metric measured spaces and also from oscill…
Study of linear classifiers in infinite imbalance scenarios.
problem Behavior of linear discriminant functions in extreme imbalance conditions.
method Analysis of linear classifiers under infinite imbalance, focusing on weight function properties and limit behavior.
result Limiting coefficient vectors reflect robustness or conservatism, optimizing against worst-case alternatives.
For any m > 0, we construct properly embedded minimal surfaces in H^2 x R with genus zero, infinitely many vertical planar ends and m limit ends. We also provide examples with an infinite countable number of limit ends. All these examples are vertical bi-graphs.
Study of deep linear neural networks with proportional width and depth.
problem Lack of descriptive power in Gaussian limit of deep linear neural networks.
method Proportional infinite-width infinite-depth limit for deep linear neural networks.
result Characterization of limiting distribution as a nontrivial mixture of Gaussians.
The paper examines how deep linear neural networks behave as they become infinitely wide.
problem Understanding the behavior of deep linear neural networks as they approach infinite width.
method Analyzes the infinite-width limit of deep linear neural networks, proving convergence to deterministic models and providing precise laws for random weights.
result The training dynamics of deep linear neural networks converge to those of a deterministic model, and the weights' behavior is precisely described.
Veech groups uniformize Teichmüller geodesic curves in Riemann moduli space. Recently, examples of infinitely generated Veech groups have been given. We show that these can even have infinitely many cusps and infinitely many infinite ends. We further show that examples exist for which each direction of an infinite end …
Paper studies Hausdorff dimensions of specific limit sets for groups on curved spaces.
problem Determining Hausdorff dimensions of non-conical and Myrberg limit sets.
method Developed techniques to calculate Hausdorff dimensions for groups acting on negatively curved spaces.
result Established maximality of Hausdorff dimension for various cases.
New framework connects two neural network theories, improving finite-width approximations.
problem Theoretical guarantees for neural network training in general cases.
method Developed a general framework linking mean-field and constant kernel theories.
result Discrete-time MF limit provides better approximation for finite-width nets.
Extends Dirac structures to infinite dimensions, focusing on convenient Lie algebroids and manifolds.
problem Extending classical geometrical results from finite to infinite dimensions.
method Introduces partial Dirac structures on convenient Lie algebroids and manifolds, explores their properties and limits.
result Classical geometrical results can be extended to infinite dimensional contexts.
We prove the existence of nonperiodic, properly embedded minimal surfaces in R2×S1 with genus zero, infinitely many ends and one limit end (in particular, they have infinite total curvature).
The paper extends Nambu-Poisson structures to infinite dimensions.
problem Extending Nambu-Poisson structures to infinite dimensional settings.
method Adapting finite dimensional Nambu-Poisson structures to a convenient manifold setting.
result Classical results in finite dimensions can be extended to infinite dimensions for partial Nambu structures.
Lectures on deep learning properties in infinite and large-width networks.
problem Understanding deep neural networks in extreme width conditions.
method Analysis of random deep neural networks, connections to linear models, kernels, and Gaussian processes, perturbative and non-perturbative treatments.
result Properties and behaviors of deep neural networks in the infinite-width limit and large-width regime.
Deep neural networks' infinite-width behavior approximated by Gaussian models.
problem Understanding the behavior of deep neural networks in the limit of infinite width.
method Using the Lindeberg exchange principle to approximate weights by Gaussian random variables.
result Quantitative bounds on the 2-Wasserstein distance between deep neural networks and Gaussian limits.
We construct a infinite-dimensional manifold structure adapted to analytic Lie pseudogroups of infinite type. More precisely, we prove that any isotropy subgroup of an analytic Lie pseudogroup of infinite type is a regular infinite-dimensional Lie group, modelled on a locally convex strict inductive limit of Banach spa…
We characterize conjugate nonparametric Bayesian models as projective limits of conjugate, finite-dimensional Bayesian models. In particular, we identify a large class of nonparametric models representable as infinite-dimensional analogues of exponential family distributions and their canonical conjugate priors. This c…
We show that the limiting unicolored sl(N) Khovanov-Rozansky chain complex of any infinite positive braid categorifies a highest-weight projector. This result extends an earlier result of Cautis categorifying highest-weight projectors using the limiting complex of infinite torus braids. Additionally, we sh…
Adaptive kernels from neural networks improve model performance.
problem Improving neural network performance through adaptive kernels.
method Deriving adaptive kernels from infinite-width neural networks using feature learning and gradient flow training.
result Adaptive kernels achieve lower test loss compared to traditional kernels.
We provide examples of finitely generated infinite covolume subgroups of PSL(2,R)r with a "big" limit set, e.g. that contains an open subset of the geometric boundary. They are given by the so called semi-arithmetic Fuchsian groups admitting modular embeddings.
Survey on Nambu-Poisson structures in infinite dimensions.
problem Generalization of Poisson and Nambu-Poisson structures in infinite dimensions.
method Study properties of associated characteristic distribution and projective/direct limits.
result Properties and limits of Nambu-Poisson structures in convenient setting.
New examples of manifolds with positive scalar curvature and infinitely many poles.
problem Constructing manifolds with positive scalar curvature and understanding their limits.
method Extending previous examples to create new sequences of manifolds.
result Found new examples of manifolds with infinitely many poles and positive scalar curvature.
New framework resolves central limit behavior in differential privacy.
problem Choosing appropriate privacy metrics in hypothesis testing.
method Infinitely divisible limit experiments and Le Cam's theory.
result Characterizes all limiting baseline trade-off functions in differential privacy.
This paper shows how infinitely wide Tensor Networks converge to Gaussian Processes.
problem Understanding the relationship between Tensor Networks and Gaussian Processes.
method Analyzing the infinite-width limit of Tensor Networks and comparing them to Gaussian Processes.
result Infinitely wide Tensor Networks converge to Gaussian Processes, proving their equivalence.
Introduces new limit spaces for degenerating Calabi-Yau families.
problem Understanding degenerating Calabi-Yau families and their limit structures.
method Introduces galaxy spaces as dense subspace of infinite open Calabi-Yau varieties.
result Galaxy spaces are projective limits of toroidal compactifications.
We construct a complete, embedded minimal surface in euclidean 3-space which has unbounded Gaussian curvature. It has infinite genus, infinitely many catenoidal type ends and one limit end.
Study of deep Stable neural networks with various activation functions.
problem Characterizing the infinitely wide limits of deep Stable neural networks.
method Investigation of large-width properties of deep Stable NNs with a generalized central limit theorem for heavy tails.
result Extension of characterization to a broader class of activation functions, including sub-linear, asymptotically linear, and super-linear functions.
Let (W,S) be a finite rank Coxeter system with W infinite. We prove that the limit weak order on the blocks of infinite reduced words of W is encoded by the topology of the Tits boundary of the Davis complex X of W. We consider many special cases, including W word hyperbolic, and X with isolated flats. We establish tha…
We generalize Bonahon's characterization of geometrically infinite torsion-free discrete subgroups of PSL(2, C) to geometrically infinite discrete isometry subgroups in the case of rank 1 symmetric spaces, and, under the assumption of bounded torsion, to the case of negatively pinched Hadamard manifolds. Eve…
Theory explains neural network scaling with dataset and model size.
problem Neural network scaling laws with dataset and model size.
method Identified variance-limited and resolution-limited scaling behaviors.
result Four scaling regimes explained: infinite data, infinite width, resolution-limited, and large width.
Theoretical limits of deep residual networks show consistent covariance structures.
problem Understanding the limits of deep residual networks.
method Analyzing the behavior of deep residual networks with skip connections as width and depth approach infinity.
result Theoretical analysis confirms that the covariance structure remains consistent regardless of the order of width and depth.
New infinite-type loxodromic elements found in surface mapping classes.
problem Identifying infinite-type loxodromic elements in mapping classes of surfaces.
method Constructing infinite families of mapping classes acting loxodromically on the relative arc graph.
result Explicit construction and characterization of infinite-type loxodromic elements.
This paper extends Lie algebra contractions to infinite-dimensional spaces for better understanding of group limits.
problem Understanding group limits through Lie algebra contractions, especially in infinite-dimensional settings.
method Using infinite-dimensional Lie algebras and their integration theory, the paper constructs Lie group expansions.
result Explicit descriptions of Lie groups in elementary terms, including applications to Newtonian gravity.
Study of deep neural networks with dependent weights leading to new model limits and properties.
problem Characterizing deep neural networks with dependent weights in the infinite-width limit.
method Modeling weights as a mixture of Gaussian distributions and analyzing the infinite-width limit.
result Characterization of neural network layers by scalar parameters and Lévy measures, leading to new model limits.
The paper explores infinite-dimensional nonholonomic and vakonomic systems.
problem Understanding dynamics of infinite-dimensional systems with constraints.
method Visualizing and revisiting classical and new examples of nonholonomic and vakonomic systems.
result Infinite-dimensional systems exhibit both nonholonomic and vakonomic dynamics.
We study the limits of sequences of spheres and complex projective spaces with unbounded dimensions. A sequence of spheres (resp. complex projective spaces) either is a Levy family, infinitely dissipates, or converges to (resp. the Hopf quotient of) a virtual infinite-dimensional Gaussian space, depending on the size o…
We aim to construct the optimal solutions to the undiscounted continuous-time infinite horizon optimization problems, the objective functionals of which may be unbounded. We identify the condition under which the limit of the solutions to the finite horizon problems is optimal for the infinite horizon problems under th…