arXiv research

New findings on depth vs. width in neural networks, showing depth can improve learnability.

Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.

The paper proves barriers to approximating functions with small weights and depth in neural networks.

Proposes a new method to estimate Bayesian neural network depth.

Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.

Bayesian linear networks reveal optimal depth and width trade-offs.

Theoretical limits of deep residual networks show consistent covariance structures.

Study infinite-depth limits of neural networks with fixed width.

One-shot neural architecture search allows joint learning of weights and network architecture, reducing computational cost. We limit our search space to the depth of residual networks and formulate an analytically tractable variational objective that allows for obtaining an unbiased approximate posterior over depths in…

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

New algorithm learns random neural networks efficiently.

Over-parameterized CNNs show U-shaped test risk with depth increase.

Study how depth affects inference in deep Bayesian neural networks.

Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and …

New findings show depth is more important than width in neural networks.

Inverse depth scaling found in LLMs due to similar layers averaging error.

Estimates neural network error approximating compact sets.

UDN adapts depth to data complexity, outperforming standard neural networks.

This paper explores how neural network width and depth behave as they approach infinity.

Monotone neural networks can approximate and interpolate functions efficiently.

Existing depth separation results for constant-depth networks essentially show that certain radial functions in $\mathbb{R}^d$, which can be easily approximated with depth $3$ networks, cannot be approximated by depth $2$ networks, even up to constant accuracy, unless their size is exponential in $d$. However, the func…

Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question:…

We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be…

Sharp representation theorems show depth benefits for ReLU networks.

The paper examines how the angle between inputs in ReLU networks decreases with depth, impacting training.

Wide neural networks converge to Gaussian processes, improving generalization.

Adversarial examples in deep ReLU networks with constant depth.

Study on neural network initialization with shaped infinite depth-and-width networks.

Study of deep linear neural networks with proportional width and depth.

Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality d…

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

The paper explores how the depth of neural networks affects their ability to represent data accurately.

Proves necessity of at least log2(n) layers to compute maximum of n numbers.

Improved neural network depth-width trade-offs via dynamical systems.

We survey results on neural network expressivity described in "On the Expressive Power of Deep Neural Networks". The paper motivates and develops three natural measures of expressiveness, which all display an exponential dependence on the depth of the network. In fact, all of these measures are related to a fourth quan…

Smooth activations enable optimal error rates in neural networks for Sobolev function classes.

Unified theory of deep neural networks with diverse activations.

Let $f:\mathbb{S}^{d-1}\times \mathbb{S}^{d-1}\to\mathbb{S}$ be a function of the form $f(\mathbf{x},\mathbf{x}') = g(\langle\mathbf{x},\mathbf{x}'\rangle)$ for $g:[-1,1]\to \mathbb{R}$. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate $f…

We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-lin…

Researchers quantify the relationship between feature depth and performance in deep neural networks.

This paper optimizes ReLU networks for approximating Hölder continuous functions.

Deep neural networks perform well on local tasks but struggle with global tasks.

Study of infinitely deep but narrow neural networks using NTK theory.

Deep RNNs excel at capturing long-term dependencies in sequential data.

Deep networks improve by progressively refining approximations at each layer.

A new data-level recombination strategy improves RGB-D salient object detection.