A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.
problem Characterizing the large-width behavior of ReLU neural networks with α-Stable initializations.
method Analysis of the large-width distributions and training dynamics of ReLU neural networks initialized with α-Stable distributions.
result For ReLU neural networks with α-Stable initializations, the large-width training dynamics achieve zero training error at a linear rate, characterized by a random kernel.
Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dyn…
Lectures on deep learning properties in infinite and large-width networks.
problem Understanding deep neural networks in extreme width conditions.
method Analysis of random deep neural networks, connections to linear models, kernels, and Gaussian processes, perturbative and non-perturbative treatments.
result Properties and behaviors of deep neural networks in the infinite-width limit and large-width regime.
We extend the classical definition of {\it width} to higher dimensional, smooth codimension 2 knots and show in each dimension there are knots of arbitrarily large width.
Study of deep Stable neural networks with various activation functions.
problem Characterizing the infinitely wide limits of deep Stable neural networks.
method Investigation of large-width properties of deep Stable NNs with a generalized central limit theorem for heavy tails.
result Extension of characterization to a broader class of activation functions, including sub-linear, asymptotically linear, and super-linear functions.
Given a 2-dimensional surface M and a constant C we construct a Riemannian metric g, so that diameter diam(M,g)=1 and every 1-cycle dividing M into two regions of equal area has length >C. It follows that there exists no universal inequality bounding 1-width of M in terms of its diameter. This answers a question of Ste…
The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which e…
A very simple R3 realization of the Möbius strip, significantly simpler than the common one, is given. For any, however large width/length ratio of the strip, it is shown that this realization, in contrast with the common one, is the union of a vertical segment and the graph of a simple rational function on …
We revisit the initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. As a consequence, this new tool ena…
In this work we construct a sequence of Riemannian metrics on the three-sphere with scalar curvature greater than or equal to 6 and arbitrarily large widths. Our procedure is based on the connected sum construction of positive scalar curvature metrics due to Gromov and Lawson. We develop analogies between the area of…
Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency --- a shallow network of exponentially lar…
Bayesian neural networks struggle with accuracy and uncertainty quantification in complex models.
problem Challenges in achieving high predictive performance and reliable uncertainty estimates in Bayesian neural networks.
method Investigates computational costs, accuracy, and uncertainty quantification in Bayesian neural networks with different inference techniques.
result Variational inference provides better uncertainty quantification than Markov chain Monte Carlo, and stacking/ensembling variational approximations can achieve similar accuracy at reduced cost.
This work finds a point with small test error in polynomial time for mildly overparameterized neural nets.
problem Achieving small test error in mildly overparameterized neural networks.
method The work shows that the landscape of loss functions with explicit regularization has a property that all local minima and certain stationary points achieve small test error. It also proves the existence of polynomial time algorithms for finding such points in convolutional and fully connected neural nets.
result Polynomial time algorithms exist for finding points with small test error in mildly overparameterized neural nets.
It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from lo…
We develop a mathematically rigorous framework for multilayer neural networks in the mean field regime. As the network's widths increase, the network's learning trajectory is shown to be well captured by a meaningful and dynamically nonlinear limit (the \textit{mean field} limit), which is characterized by a system of …