arXiv research

Estimates neural network error approximating compact sets.

New findings on depth vs. width in neural networks, showing depth can improve learnability.

This paper studies the expressive power of graph neural networks falling within the message-passing framework (GNNmp). Two results are presented. First, GNNmp are shown to be Turing universal under sufficient conditions on their depth, width, node attributes, and layer expressiveness. Second, it is discovered that GNNm…

Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.

Study infinite-depth limits of neural networks with fixed width.

Theoretical limits of deep residual networks show consistent covariance structures.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

Improved neural network depth-width trade-offs via dynamical systems.

New findings show depth is more important than width in neural networks.

This paper explores how neural network width and depth behave as they approach infinity.

Unified spectral framework for μP under joint width-depth scaling.

Deep forests enhance expressiveness exponentially with depth, not width or tree size.

ResNets approximate log-Gaussian at initialization, improving network performance.

Bayesian linear networks reveal optimal depth and width trade-offs.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Complex-valued neural networks can approximate any continuous function with bounded widths and depths.

New study shows neural networks can generalize without gradient descent, especially in deep settings.

Study on neural network initialization with shaped infinite depth-and-width networks.

Study of deep linear neural networks with proportional width and depth.

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d_{in}$, hidden layer widths at most $w,$ and …

We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets. Our analysis reveals that finite size residual architectures are initialized much closer to the "kernel regime" than their vanilla counterparts: while in networks that do not use skip connections, convergence to t…

New insights into how depth and width affect in-context learning in deep models.

This paper optimizes ReLU networks for approximating Hölder continuous functions.

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N…

New Transformer architecture prevents rank degeneracy in deep attention models.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

This paper improves neural network approximation for analytic functions with adjustable depth and width.

Adversarial examples in deep ReLU networks with constant depth.

Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.

This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.

We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-lin…

Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory. In a seminal paper, Telgarsky highlig…

The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let $n$ be the number of inputs neurons, $m$ be the number of output neurons, and let $ρ$ be any nonaff…

WGANs improve probability distribution approximation with depth and width trade-offs.

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local…

New neural network architecture with height adds expressive power.

Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…

Deep networks can perfectly classify two low-dimensional manifolds on a sphere with large depth and width.

Wide neural networks can degrade performance, contrary to conventional wisdom.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

CNNs improve generalization to unseen audio devices with increased width, not depth.

While classic studies proved that wide networks allow universal approximation, recent research and successes of deep learning demonstrate the power of deep networks. Based on a symmetric consideration, we investigate if the design of artificial neural networks should have a directional preference, and what the mechanis…

Maximal initial learning rate for deep ReLU networks identified.

Study examines dependence properties of Bayesian neural network units in finite-width networks.

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

MPNNs over-squash distant node information, study shows.

We show that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all kn…

Batch normalization makes deep neural networks' representations increasingly orthogonal.