arXiv research

Study infinite-depth limits of neural networks with fixed width.

Wide neural networks converge to Gaussian processes, improving generalization.

Study on neural network initialization with shaped infinite depth-and-width networks.

ResNets approximate log-Gaussian at initialization, improving network performance.

Study of infinitely deep but narrow neural networks using NTK theory.

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

Study of deep linear neural networks with proportional width and depth.

New Transformer architecture prevents rank degeneracy in deep attention models.

UDN adapts depth to data complexity, outperforming standard neural networks.

Deep ResNets favor low bottleneck rank with proper hyperparameters.

Bayesian linear networks reveal optimal depth and width trade-offs.

When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parame…

Theoretical limits of deep residual networks show consistent covariance structures.

New framework combines simple machines into complex ones for better neural network performance.

Stable ResNet stabilizes gradients in deep networks.

Generates infinite-depth hierarchical clusters from few examples.

DEQs converge to optimal solutions with mild over-parameterization.

Hypersolvers enable fast continuous-depth models for practical applications.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Propagation-regularization improves GNN performance by infusing extra graph information.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…

We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is…

Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.

New findings connect shaped and unshaped neural networks using differential equations.

Researchers dissect Neural ODEs to understand their dynamics.

Large neural networks learn low-dimensional representations that balance complexity and regularity.

New framework improves robustness of implicit neural networks.

Implicit models can match or exceed explicit models with more test-time compute.

Study examines infinite limits of transformer dynamics, identifying key parameterizations.

Study on feature learning in Leaky ResNets, explaining bottleneck structure.

FNO-DEQ solves steady-state PDEs as fixed points, outperforming traditional FNOs.

Transformers preserve support and can approximate any continuous map.

DiffEnc improves diffusion models by adding flexibility and achieving better likelihood on CIFAR-10.

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d_{in}$, hidden layer widths at most $w,$ and …

SONODEs and ANODEs improve learning of second order dynamics.

ResNets and their GP generalization align ideas via function space warping, revealing robust properties and connections to image registration.

ResNets converge to a limit model with improved error rates.

Online minimization of an unknown convex function over the interval $[0,1]$ is considered under first-order stochastic bandit feedback, which returns a random realization of the gradient of the function at each query point. Without knowing the distribution of the random gradients, a learning algorithm sequentially choo…

Continuum transformers learn operators in context via gradient descent.

We describe a construction of ordered algebraic structures (ordered abelian semigroups, ordered commutative semirings, etc.) and describe applications to codimension-1 laminations. For a suitable ordered semi- algebraic structure $\mathbb L$ and measurable space $X$ we define $\mathbb L$-measures $ν$ on $X$. If $L$ is …

RFN improves GCNs for road networks, outperforming state-of-the-art by 21%-40%.

This survey clarifies dynamic network terminology and reviews GNN models for dynamic networks.

From the perspective of network analysis, the ubiquitous networks are comprised of regular and irregular components, which makes uncovering the complexity of network structures to be a fundamental challenge. Exploring the regular information and identifying the roles of microscopic elements in network data can help us …

Chemical networks outperform spiking neural networks in classification tasks.

Deep ReLU networks can be simplified to a three-layer model.

This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.

New approach learns latent motifs in networks for mesoscale structure analysis.