arXiv research

Lectures on deep learning properties in infinite and large-width networks.

This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.

The paper studies deep neural networks with Gaussian weights and finds their asymptotic behavior.

NTK theory fails to predict practical behavior of large-width neural networks.

Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dyn…

This work studies fluctuation in multilayer neural networks using mean field theory.

This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.

Wide neural networks can degrade performance, contrary to conventional wisdom.

Theory explains neural network scaling with dataset and model size.

Study of deep Stable neural networks with various activation functions.

Elman-type RNNs converge to globally optimal solutions in the mean-field regime.

We extend the classical definition of {\it width} to higher dimensional, smooth codimension 2 knots and show in each dimension there are knots of arbitrarily large width.

This paper analyzes deep Stable neural networks, showing convergence rates under different growth settings.

Randomly initialized wide neural networks with zero-mean activations are nearly independent, potentially solving AI interpretability limits.

Wide networks with polynomial activations have proven asymptotic behavior.

Improved LoRA+ adapts large models more efficiently.

Adaptive kernels from neural networks improve model performance.

Given a 2-dimensional surface M and a constant C we construct a Riemannian metric g, so that diameter diam(M,g)=1 and every 1-cycle dividing M into two regions of equal area has length >C. It follows that there exists no universal inequality bounding 1-width of M in terms of its diameter. This answers a question of Ste…

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Quantum neural tangent kernels help understand variational quantum circuits in machine learning.

DNNs improve SIMP method but not spatially invariant, study shows.

A very simple $\mathbb R^3$ realization of the Möbius strip, significantly simpler than the common one, is given. For any, however large width/length ratio of the strip, it is shown that this realization, in contrast with the common one, is the union of a vertical segment and the graph of a simple rational function on …

GNTK reveals convergence of GNNs on large graphs.

Weak correlations explain linear dynamics in deep learning models.

Global convergence proved for three-layer neural networks in mean field regime.

Researchers derive exact priors for finite Bayesian neural networks.

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which e…

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

In this work we construct a sequence of Riemannian metrics on the three-sphere with scalar curvature greater than or equal to $6$ and arbitrarily large widths. Our procedure is based on the connected sum construction of positive scalar curvature metrics due to Gromov and Lawson. We develop analogies between the area of…

3-manifolds with torsion homology expand in all dimensions.

Study of deep neural networks with dependent weights leading to new model limits and properties.

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

New approach predicts generalization of deep neural networks in proportional-width regime.

We develop a mathematically rigorous framework for multilayer neural networks in the mean field regime. As the network's widths increase, the network's learning trajectory is shown to be well captured by a meaningful and dynamically nonlinear limit (the \textit{mean field} limit), which is characterized by a system of …

It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from lo…

This paper improves the neural network-QFT correspondence by nonperturbative renormalization.

The study examines spectral dynamics in deep neural networks, predicting how outliers evolve during training.

Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency --- a shallow network of exponentially lar…

Noise injection before gradient steps helps in regularization for neural networks.

This paper uses second-order Poincaré inequalities to establish quantitative central limit theorems for Gaussian neural networks.

Linearized attention fails to converge to NTK limit even at large widths.

This paper studies bounds for the Lipschitz constant of random neural networks.

Study shows how large neural networks avoid overfitting through decoupling of feature learning and complexity growth.

We revisit the initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. As a consequence, this new tool ena…

This work finds a point with small test error in polynomial time for mildly overparameterized neural nets.

Early neural networks can be simplified to linear models, revealing surprising simplicity.

DNNs can learn complex functions efficiently by breaking the curse of dimensionality.

A ribbon is, intuitively, a smooth mapping of an annulus $S^1 \times I$ in 3-space having constant width $\varepsilon$. This can be formalized as a triple $(x,\varepsilon, \mathbf{u})$ where $x$ is smooth curve in 3-space and $\mathbf{u}$ is a unit vector field based along $x$. In the 1960s and 1970s, G. Calugareanu, G…