Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count% of corpus
0.3%0.5%0.8%0.1% · Jun 202219922001200920172026
4 results for “infinite-depth-and-width”
The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationstat.ML

ResNets approximate log-Gaussian at initialization, improving network performance.

problem Understanding the initialization behavior of deep neural networks like ResNets.
method Analyzing ReLU ResNets in the infinite-depth-and-width limit, showing log-Gaussian behavior.
result ResNets at initialization exhibit hypoactivation and interlayer correlations, which are not captured by Gaussian limits.
ResNetslog-Gaussianinfinite-depth-and-widthhypoactivationinterlayer correlations
2021-06-07abs ↗pdf ↗
The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationstat.ML

Study on neural network initialization with shaped infinite depth-and-width networks.

problem Understanding the distribution of random covariance matrices in shaped infinite-depth-and-width networks.
method Introduced the Neural Covariance SDE to model the distribution of the random covariance matrix.
result Identified the precise scaling of the activation function necessary for a non-trivial limit.
neural networksinfinite depthinfinite widthcovariance matrixstochastic differential equation
2022-06-06abs ↗pdf ↗
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitstat.ML

New Transformer architecture prevents rank degeneracy in deep attention models.

problem Rank degeneracy in deep attention models.
method Modified Softmax-based attention model with skip connections, centered at identity, and scaled logits.
result Existence of a stable SDE implies well-behaved covariance structure, preventing rank degeneracy.
Transformerattention modelsinfinite depth-and-widthcovariance matrixstochastic differential equation
2023-06-30abs ↗pdf ↗
Differential Equation Scaling Limits of Shaped and Unshaped Neural Networksstat.ML

New findings connect shaped and unshaped neural networks using differential equations.

problem Understanding the behavior of neural networks with different activation scaling methods.
method Deriving differential equation-based asymptotic characterizations for shaped and unshaped neural networks.
result Two types of unshaped networks converge to the same infinite-depth-and-width limit at initialization.
neural networksdifferential equationsactivation scalingshaped networksunshaped networks
2023-10-18abs ↗pdf ↗