arXiv research

Develops a new theory for neural systems stability and width effects.

Wide neural networks can degrade performance, contrary to conventional wisdom.

Fisher width is a geometric measure of complexity on statistical manifolds.

The study reveals a transition in neural network performance from infinite-width to variance-limited behavior as dataset size increases.

Study on spherical bodies of constant width on the unit sphere, proving bounds on their relative effective radius.

Study on fluctuations in neural network kernels and predictions, focusing on finite width effects.

Deep networks can be biased to learn top eigenfunctions of the kernel outside the training set.

Wider neural networks perform better than deeper ones with the same number of parameters.

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

New approach predicts generalization of deep neural networks in proportional-width regime.

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local…

ResNets approximate log-Gaussian at initialization, improving network performance.

Stochastic neural networks with infinite width become deterministic, reducing training variance.

Paper analyzes infinite-width attention layers using Tensor Programs.

Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Bayesian linear networks reveal optimal depth and width trade-offs.

Unified spectral framework for μP under joint width-depth scaling.

We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…

Bayesian Bits unifies quantization and pruning through gradient optimization.

Sharp feature transitions revealed in extensive-width networks.

We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.

Wide Bayesian neural networks have a simpler weight posterior, leading to faster MCMC sampling.

New analysis shows optimal embedding learning rate depends on vocabulary size, not just model width.

Study apple tasting feedback in online binary classification, providing new insights into minimax expected mistakes.

This work optimizes neural network bit-width and layer-width for efficiency.

We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, a…

We study the effect of the social stratification on the wealth distribution on a system of interacting economic agents that are constrained to interact only within their own economic class. The economical mobility of the agents is related to its success in exchange transactions. Different wealth distributions are obtai…

Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to in…

Paper characterizes gradient descent dynamics for neural networks with finite width.

We define the Wirtinger width of a knot. Then we prove the Wirtinger width of a knot equals its Gabai width. The algorithmic nature of the Wirtinger width leads to an efficient technique for establishing upper bounds on Gabai width. As an application, we use this technique to calculate the Gabai width of approximately …

Derives a family of hyperparameter scaling strategies for neural networks.

Improves matrix multiplication throughput for asymmetric bit-width operands.

Empirical study compares finite- and infinite-width BNNs, revealing performance differences under model mismatch.

Effective theory for Transformer initialization improves model performance.

The isospectral problem for p-widths is solved using Zoll metrics on S^2.

Neural network-based post-processing improves ensemble forecast sharpness

Width trees link link invariants and bridge number.

Study on neural network dynamics in high dimensions with quadratic activation.

Lectures on deep learning properties in infinite and large-width networks.

Computed p-widths for hemisphere, first for manifolds with boundary.

Polygon $p$-widths are found via billiard trajectories.

A number of results for C$^2$-smooth surfaces of constant width in Euclidean 3-space ${\mathbb{E}}^3$ are obtained. In particular, an integral inequality for constant width surfaces is established. This is used to prove that the ratio of volume to cubed width of a constant width surface is reduced by shrinking it along…

Computed p-widths for real projective plane.

Study bounds Urysohn width of manifolds under surgeries.

A remarkable characteristic of overparameterized deep neural networks (DNNs) is that their accuracy does not degrade when the network's width is increased. Recent evidence suggests that developing compressible representations is key for adjusting the complexity of large networks to the learning task at hand. However, t…

Residual networks with block width max(d_x, d_y) approximate all functions.

We study the number of Darboux charts needed to cover a closed connected symplectic manifold $(M,ω)$, and effectively estimate this number from below and from above in terms of the Lusternik--Schnirelmann category of $M$ and the Gromov width of $(M,ω)$.