Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
Study examines dependence properties of Bayesian neural network units in finite-width networks.
problem Understanding dependence properties of hidden units in practical finite-width Bayesian neural networks.
method Theoretical analysis and empirical evaluation of depth and width impacts.
result Hidden units in finite-width Bayesian neural networks are dependent, contrary to the infinite-width limit assumption.
Study on hidden units in finite Bayesian neural networks and their tail properties.
problem Understanding the behavior of hidden units in finite Bayesian neural networks.
method Introduced a generalized Weibull-tail property to describe hidden units tails.
result Unit priors become heavier-tailed going deeper, providing insights into finite Bayesian neural networks.
New surfaces show horocyclic flow isn't always minimal.
problem Complex dynamics on infinite fineness surfaces.
method Construction of infinite hyperbolic surfaces.
result Horocyclic flow is not minimal on infinite fineness surfaces.
Simplified and extended a method for rearranging infinite configurations of cubes.
problem Constructing homotopies for isotopically rearranging cubes.
method Simplified and extended Eda and Kawamura's procedure.
result Simplified and extended the construction of homotopies.
We show an energy convexity along any harmonic map heat flow with small initial energy and fixed boundary data on the unit 2-disk. In particular, this gives an affirmative answer to a question raised by W. Minicozzi asking whether such harmonic map heat flow converges uniformly in time strongly in the W^{1,2}-topology,…
Study on optimal ReLU networks with weight decay for interpolation.
problem Interpolating data with radially symmetric distributions using shallow ReLU networks.
method Weight decay regularization in infinite neuron, infinite data limit; analysis of growth rates.
result Existence and growth rates of unique radially symmetric minimizers with weight decay.
Book on infinite-dimensional Lie groups, covering basics and various classes.
problem Understanding Lie groups in infinite-dimensional spaces.
method Develops smooth manifolds and Lie groups in locally convex spaces, discussing various classes.
result Detailed exploration of infinite-dimensional Lie groups and their properties.
Study constructs disks with curved boundaries in a 3D ball.
problem Constructing non-planar free boundary disks in a unit ball.
method Infinite family of non-planar disks with non-positive Gaussian curvature.
result Constructs disks with curved boundaries in a unit ball.
We study the properly discontinuous and isometric actions on the unit sphere of infinite dimensional Hilbert spaces and we get some new examples of Hilbert manifold with costant positive sectional curvature. We prove some necessary conditions for a group to act isometrically and properly discontinuously and in the case…
This paper extends ResNet theory to infinitely deep networks, linking them to diffusion processes.
problem Training infinitely deep ResNets with i.i.d. initializations leads to undesirable properties.
method Introduced doubly infinite ResNets with i.i.d. initializations, linking to diffusion processes.
result The dynamics of quantities of interest converge to deterministic limits in the limit of infinite depth.
Mathematical methods characterize RNNs' asymptotics as hidden units and data grow.
problem Characterize recurrent neural networks' behavior as hidden units and data grow.
method Developed mathematical methods to analyze RNNs' convergence to an infinite-dimensional ODE coupled with a fixed point of a random algebraic equation.
result RNNs converge to an infinite-dimensional ODE coupled with a fixed point of a random algebraic equation.
The infinite restricted Boltzmann machine (iRBM) is an extension of the classic RBM. It enjoys a good property of automatically deciding the size of the hidden layer according to specific training data. With sufficient training, the iRBM can achieve a competitive performance with that of the classic RBM. However, the c…
Study examines infinite limits of transformer dynamics, identifying key parameterizations.
problem Understanding the training dynamics of transformer models in the feature learning regime.
method Analysis of infinite scaling limits using dynamical mean field theory.
result Identified parameterizations that admit well-defined infinite width and depth limits.
Study shows orbits on a specific surface without intersecting geodesics.
problem Understanding orbits on a specific surface without intersecting geodesics.
method Analyzing the horocyclic flow on the unit tangent bundle of an untwisted flute.
result Recurrent and irregular orbits do not intersect closed geodesics.
Smooth groupoid algebras are H-unital, with implications for algebraic and homological properties.
problem Understanding the structure of convolution algebras on Lie groupoids.
method Analyzing smooth functions and invariant subsets to prove H-unitality.
result H-unitality of groupoid algebras and their quotients, leading to excision properties.
Study infinite circle patterns in the Weil-Petersson class using discrete harmonic functions.
problem Characterize infinite circle patterns in the Weil-Petersson class.
method Investigate circle patterns parameterized by discrete harmonic functions of finite Dirichlet energy, equipped with a Riemannian metric.
result Induced quasiconformal homeomorphisms from the unit disk to itself belong to the Weil-Petersson class.
New kernels from ELU and GELU networks reveal non-trivial fixed points.
problem Understanding fixed-point dynamics in deep neural networks with ELU and GELU activations.
method Deriving covariance functions and analyzing fixed-point dynamics of ELU and GELU networks.
result ELU and GELU networks exhibit non-trivial fixed-point dynamics, explaining implicit regularization in overparameterized models.
This work presents a parametrized family of divergences, namely Alpha-Beta Log- Determinant (Log-Det) divergences, between positive definite unitized trace class operators on a Hilbert space. This is a generalization of the Alpha-Beta Log-Determinant divergences between symmetric, positive definite matrices to the infi…
Study infinite-depth limits of neural networks with fixed width.
problem Understanding the behavior of neural networks as depth increases with fixed width.
method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.
result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.
Newton's method converges faster than gradient descent in overparameterized neural networks.
problem Training neural networks efficiently in the overparameterized limit.
method Developed a convergence analysis for the regularized Newton method in this context.
result The NN training dynamics converge to the solution of a deterministic limit equation involving a Newton neural tangent kernel (NNTK).
This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.
problem Exact data interpolation using sparse, infinitely wide neural networks.
method Atomic norm framework to derive convex hulls and equivalent convex formulations.
result Simple characterizations of convex hulls for different constraints on network weights and biases.
We introduce a structure of an infinite-dimensional Frobenius manifold on a subspace in the space of pairs of functions analytic inside/outside the unit circle with simple poles at 0/infinity respectively. The dispersionless 2D Toda equations are embedded into a bigger integrable hierarchy associated with this Frobeniu…
The Yamabe flow can blow up in infinite time with small perturbations.
problem Understanding the behavior of the Yamabe flow under small perturbations.
method Constructive proof using solutions of the Yamabe problem on the unit sphere as blow-up profiles.
result The Yamabe flow can blow up at multiple points on a Riemannian manifold in infinite time with small perturbations.
A left orderable completely metrizable topological group is exhibited containing Artin's braid group on infinitely many strands. The group is the mapping class group (rel boundary) of the closed unit disk with a sequence of interior punctures converging to the boundary. This resolves an issue suggested by work of Dehor…
Infinite CNNs lose spatial correlations, but can be restored by correlated weights.
problem Infinite CNNs lose spatial correlations, which are crucial for their performance.
method Introduced correlated weights to restore spatial correlations in infinite CNNs.
result Optimal performance is achieved with a moderate level of weight correlation.
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal …
Multivariate splines linked to infinitely-wide neural networks with improved numerical performance.
problem Understanding the relationship between multivariate splines and neural networks.
method Showed multivariate splines can be represented as random features in infinitely-wide neural networks with a homogeneous activation function.
result The function space of multivariate splines is a Sobolev space on a Euclidean ball with explicit norm bounds on derivatives.
The study of universal approximation of arbitrary functions f:X→Y by neural networks has a rich and thorough history dating back to Kolmogorov (1957). In the case of learning finite dimensional maps, many authors have shown various forms of the universality of both fixed depth and fixed width…
In this article we investigate a monoid of smooth mappings on the space of arrows of a Lie groupoid and its group of units. The group of units turns out to be an infinite-dimensional Lie group which is regular in the sense of Milnor. Furthermore, this group is closely connected to the group of bisections of the Lie gro…
Geometrically infinite Kleinain groups have nonconical limit sets with the cardinality of the continuum. In this paper, we construct a geometrically infinite Fuchsian group such that the Hausdorff dimension of the nonconical limit set equals zero. For finitely generated, geometrically infinite Kleinian groups, we prove…
We construct a class of infinite-dimensional Frobenius manifolds on the space of pairs of certain even functions meromorphic inside or outside the unit circle. Via a bi-Hamiltonian recursion relation, the principal hierarchies associated to such Frobenius manifolds are found to be certain extensions of the dispersionle…
In this work, we propose an infinite restricted Boltzmann machine~(RBM), whose maximum likelihood estimation~(MLE) corresponds to a constrained convex optimization. We consider the Frank-Wolfe algorithm to solve the program, which provides a sparse solution that can be interpreted as inserting a hidden unit at each ite…
Wide neural networks can benefit from multi-task learning in their infinite-width limit.
problem The generalization behavior of wide neural networks in multi-task learning settings.
method Optimizing wide ReLU neural networks with L2-regularization promotes multi-task learning in the infinite-width limit.
result An exact quantitative characterization of multi-task learning in the infinite-width limit of wide ReLU neural networks.
We show that the limiting Khovanov chain complex of any infinite positive braid categorifies the Jones-Wenzl projector. This result extends Lev Rozansky's categorification of the Jones-Wenzl projectors using the limiting complex of infinite torus braids. We also show a similar result for the limiting Lipshitz-Sarkar-Kh…
Neural Q-learning tackles high-dimensional PDEs.
problem Solving high-dimensional PDEs is computationally challenging.
method Adapting Q-learning from reinforcement learning to solve PDEs.
result The neural network approximator converges to the PDE solution as the network width increases.
New polynomial invariants for knots and links.
problem Defining new invariants for knot theory.
method Infinite family of quiver representations.
result Infinite family of two-variable polynomial invariants.
Researchers derive exact priors for finite Bayesian neural networks.
problem Understanding non-Gaussian priors in finite Bayesian neural networks.
method Analytical derivation of function space priors for finite fully-connected feedforward networks.
result Exact solutions for priors of finite networks, including Meijer G-function for linear networks and mixtures for ReLU networks.
An embedding of the group $\Diff(S^{1})$ of orientation preserving diffeomorphims of the unit circle S1 into an infinite-dimensional symplectic group, $\Sp(\infty)$, is studied. The authors prove that this embedding is not surjective. A Brownian motion is constructed on $\Sp(\infty)$. This study is motivated by rece…
New framework for understanding infinite-width neural networks.
problem Understanding the infinite-width limit behavior of neural networks.
method General framework to study limit behavior of neural models based on hyperparameter scaling.
result Derives scaling for existing mean-field and neural tangent kernel limits and introduces new dynamically stable limits.
New examples of harmonic unit vector fields on hyperbolic 3-space are constructed by exploiting the reduction of symmetry arising from the foliation by horospheres. This is compared and contrasted with the analogous construction in Euclidean 3-space, using a foliation by planes, which produces some new examples of harm…
Study on geodesics and dihedral groups in lattices.
problem Growth and distribution of conjugacy classes of dihedral subgroups.
method Generalizing earlier work on reciprocal geodesics, proving equidistribution.
result Reciprocal geodesics are equidistributed in the unit tangent bundle.
Paper analyzes infinite-width attention layers using Tensor Programs.
problem Capturing the infinite-width limit of attention layers.
method Tensor Programs framework to rigorously identify the limit distribution.
result Derives exact form of infinite-width limit distribution without Gaussian approximations.
We show that there are infinitely many nonisomorphic quandle structures on any topogical space X of positive dimension. In particular, we disprove the conjecture, asserting that there are no nontrivial quandle structures on the closed unit interval [0,1].
We make systematic developments on Lawson-Osserman constructions relating to the Dirichlet problem (over unit disks) for minimal surfaces of high codimension in their 1977 Acta paper. In particular, we show the existence of boundary functions for which infinitely many analytic solutions and at least one nonsmooth Lipsc…
Bayesian neural networks approximate Student-t processes in the infinite-width limit.
problem Modeling uncertainty in neural networks with greater flexibility.
method Extending asymptotic properties of Gaussian processes to Student-t processes in the infinite-width limit of BNNs.
result Posterior BNNs converge to Student-t processes in the infinite-width limit.
Infinite G-invariant minimal hypersurfaces found in Riemannian manifolds.
problem Finding minimal hypersurfaces in manifolds with group actions.
method New algorithm using multi-stage maximal cuttings.
result Each G-homology class admits infinitely many distinct realizations by embedded minimal G-hypersurfaces. Study transverse measures on infinite type hyperbolic surfaces.
problem Characterize the cone of transverse measures on infinite type hyperbolic surfaces.
method Use inverse limits and geodesic laminations to describe and construct cones of transverse measures.
result Explicit descriptions and bases of cones of transverse measures exist for many laminations.