arXiv research

Unified spectral framework for μP under joint width-depth scaling.

This paper explores how neural network width and depth behave as they approach infinity.

Theoretical limits of deep residual networks show consistent covariance structures.

Study infinite-depth limits of neural networks with fixed width.

Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.

New insights into how depth and width affect in-context learning in deep models.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

ResNets and DenseNets converge to NTK with depth and width, offering advantages for kernel regression.

New Transformer architecture prevents rank degeneracy in deep attention models.

Study on neural network initialization with shaped infinite depth-and-width networks.

Inverse depth scaling found in LLMs due to similar layers averaging error.

New scaling framework for MoE architectures ensures stability and optimal performance at scale.

This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.

Adversarial examples in deep ReLU networks with constant depth.

Pruned neural networks' error scales predictably with architecture and task.

This paper improves neural network approximation for analytic functions with adjustable depth and width.

Study how depth affects inference in deep Bayesian neural networks.

New findings on depth vs. width in neural networks, showing depth can improve learnability.

Derives a family of hyperparameter scaling strategies for neural networks.

Improved neural network depth-width trade-offs via dynamical systems.

New findings show depth is more important than width in neural networks.

CNNs improve generalization to unseen audio devices with increased width, not depth.

Deep forests enhance expressiveness exponentially with depth, not width or tree size.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

ResNets approximate log-Gaussian at initialization, improving network performance.

Bayesian linear networks reveal optimal depth and width trade-offs.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performan…

This paper studies activation sparsity in large language models, finding key trends and implications.

Estimates neural network error approximating compact sets.

Complex-valued neural networks can approximate any continuous function with bounded widths and depths.

Study of deep linear neural networks with proportional width and depth.

Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.

New findings connect shaped and unshaped neural networks using differential equations.

This paper studies the expressive power of graph neural networks falling within the message-passing framework (GNNmp). Two results are presented. First, GNNmp are shown to be Turing universal under sufficient conditions on their depth, width, node attributes, and layer expressiveness. Second, it is discovered that GNNm…

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d_{in}$, hidden layer widths at most $w,$ and …

This paper optimizes ReLU networks for approximating Hölder continuous functions.

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N…

Extends hyperparameter transfer across model sizes and modules, improving training speed.

Deep networks can approximate various activation functions with modest adjustments.

Dropout and RaM become equivalent in large ResNets as depth and width increase.

Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.

Study on learning rates in neural networks of varying depth.

Bayesian neural networks learn efficiently at infinite width, matching polynomial-width performance.

Quantitative CLTs show neural network distributions converge to Gaussian as width increases.

New approach predicts generalization of deep neural networks in proportional-width regime.