arXiv research

Study infinite-depth limits of neural networks with fixed width.

New findings on depth vs. width in neural networks, showing depth can improve learnability.

Theoretical limits of deep residual networks show consistent covariance structures.

UDN adapts depth to data complexity, outperforming standard neural networks.

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

Study on neural network initialization with shaped infinite depth-and-width networks.

Study of deep linear neural networks with proportional width and depth.

Study of infinitely deep but narrow neural networks using NTK theory.

ResNets approximate log-Gaussian at initialization, improving network performance.

Wide neural networks converge to Gaussian processes, improving generalization.

Bayesian linear networks reveal optimal depth and width trade-offs.

Study examines infinite limits of transformer dynamics, identifying key parameterizations.

New Transformer architecture prevents rank degeneracy in deep attention models.

We develop a scalable method for Bayesian neural networks with stochastic differential equations.

We describe a procedure for creating infinite families of hyperbolic knots having unique minimal genus Seifert surface. A large subset of these knots have the further property that the surface cannot be the sole compact leaf of a depth one foliation of the knot exterior.

This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Study how depth affects inference in deep Bayesian neural networks.

Generates infinite-depth hierarchical clusters from few examples.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

Hypersolvers enable fast continuous-depth models for practical applications.

Maps on infinite-type surfaces linked to 3-manifold flows.

Stable ResNet stabilizes gradients in deep networks.

We obtain new invariants of topological link concordance and homology cobordism of 3-manifolds from Hirzebruch-type intersection form defects of towers of iterated p-covers. Our invariants can extract geometric information from an arbitrary depth of the derived series of the fundamental group, and can detect torsion wh…

Deep neural networks perform well on local tasks but struggle with global tasks.

DEQs converge to optimal solutions with mild over-parameterization.

Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…

This paper explores how neural network width and depth behave as they approach infinity.

This paper extends ResNet theory to infinitely deep networks, linking them to diffusion processes.

New insights into how depth and width affect in-context learning in deep models.

GCNs distinguish graph models based on embeddings, but depth matters.

The paper examines how the angle between inputs in ReLU networks decreases with depth, impacting training.

Researchers dissect Neural ODEs to understand their dynamics.

When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parame…

Study on critical points in random neural networks, revealing three regimes based on activation function.

Deeper networks are better for local labels, but shallower for global labels.

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d_{in}$, hidden layer widths at most $w,$ and …

This paper improves neural network approximation for analytic functions with adjustable depth and width.

The Baumslag-Solitar groups: BS(m,n)=<x,y| x y^{m} x^{-1} = y^{n}> are some of the simplest interesting infinite groups which are not lattices in Lie groups. They have been studied in depth from the point of view of combinatorial group theory. It is natural to ask if the geometric approach to the theory of infinite gro…

A generic geodesic on a finite area, hyperbolic 2-orbifold exhibits an infinite sequence of penetrations into a neighborhood of a cone singularity, so that the sequence of depths of maximal penetration has a limiting distribution. The distribution function is the same for all such surfaces and is described by a fairly …

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Large neural networks learn low-dimensional representations that balance complexity and regularity.

Deep ResNets favor low bottleneck rank with proper hyperparameters.

Characterizes neural kernel and NNGP for various activations.

Paper analyzes soft tree ensembles using NTK, finding only leaf count matters.

This paper investigates the approximation power of three types of random neural networks: (a) infinite width networks, with weights following an arbitrary distribution; (b) finite width networks obtained by subsampling the preceding infinite width networks; (c) finite width networks obtained by starting with standard G…

The paper examines when NTK theory applies to real finite-width neural networks.

This paper explains GCNs using NTKs and improves their performance.