arXiv research

We study how finite Bayesian neural networks adapt their hidden representations.

Develops a new theory for neural systems stability and width effects.

Analyzes DNNs trained with noisy gradients, finding FWCs negligible for large n.

This paper investigates the approximation power of three types of random neural networks: (a) infinite width networks, with weights following an arbitrary distribution; (b) finite width networks obtained by subsampling the preceding infinite width networks; (c) finite width networks obtained by starting with standard G…

Study on fluctuations in neural network kernels and predictions, focusing on finite width effects.

Develops EFT for ResNets, revealing limitations of kernel-only approach.

Study loop corrections in random feature models affecting training and test errors.

Simplified neural network EFTs reveal a single critical condition.

Novel algorithms improve efficiency of finite width NTK computation.

The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.

Renormalization in neural networks linked to quantum field theory.

Ensembles of neural networks improve training dynamics and performance.

We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, gi…

Quantum field theory connects deep neural networks to criticality.

Study examines dependence properties of Bayesian neural network units in finite-width networks.

New framework for understanding infinite-width neural networks.

We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation (Bölcskei et al., 2018) of polynomials, windowed sinusoidal functions, one-dimensional oscillatory textures, and the Weierstrass function, a fractal function which is continuous but nowhere differentiable. Together with thei…

Empirical study compares finite- and infinite-width BNNs, revealing performance differences under model mismatch.

Empirical study compares wide neural networks to kernel methods, resolving open questions.

We analyze double descent in finite-width neural networks using influence functions.

Paper characterizes gradient descent dynamics for neural networks with finite width.

Empirical study shows standard CNNs deviate from NTK predictions.

We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets. Our analysis reveals that finite size residual architectures are initialized much closer to the "kernel regime" than their vanilla counterparts: while in networks that do not use skip connections, convergence to t…

Study infinite-depth limits of neural networks with fixed width.

Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodo…

We focus on estimating \emph{a priori} generalization error of two-layer ReLU neural networks (NNs) trained by mean squared error, which only depends on initial parameters and the target function, through the following research line. We first estimate \emph{a priori} generalization error of finite-width two-layer ReLU …

Study Gaussian-process limits of neural networks using tensor programs.

Study on Bayesian deep linear networks with multiple outputs and convolutional layers.

Wide networks with polynomial activations have proven asymptotic behavior.

New framework analyzes deep learning optimization with finite width networks, revealing generalization gaps and excess risks.

Gradient descent converges linearly in finite-width networks with positive NTK and compatible conditions.

Study shows neural networks trained with GD converge to Gaussian processes with polynomial decay.

This is an exposition of aspects of the result of Daskalopoulos and Sesum that any 2-dimensional complete noncompact ancient solution to Ricci flow with bounded positive scalar curvature and finite width must be the cigar soliton.

We propose \emph{Taylorized training} as an initiative towards better understanding neural network training at finite width. Taylorized training involves training the $k$-th order Taylor expansion of the neural network at initialization, and is a principled extension of linearized training---a recently proposed theory …

Paper studies ResNet dynamics using NTH, reducing width requirement.

Study on hidden units in finite Bayesian neural networks and their tail properties.

This work establishes the equivalence between neural networks and support vector machines.

We show that a finite collection of stable subgroups of a finitely generated group has finite height, finite width and bounded packing. We then use knowledge about intersections of conjugates to characterize finite families of quasimorphisms on hyperbolically embedded subgroups that can be to simultaneously extended to…

The paper provides non-asymptotic Edgeworth expansions for neural network outputs.

Let $M$ be a Riemannian manifold with a smooth boundary. The main question we address in this article is: "When is the Laplace-Beltrami operator $Δ\colon H^{k+1}(M)\cap H^1_0(M) \to H^{k-1}(M)$, $k\in \mathbb{N}_0$, invertible?" We consider also the case of mixed boundary conditions. The study of this main question lea…

Wide CNNs outperform infinite width networks, revealing scaling laws.

There are currently two parameterizations used to derive fixed kernels corresponding to infinite width neural networks, the NTK (Neural Tangent Kernel) parameterization and the naive standard parameterization. However, the extrapolation of both of these parameterizations to infinite width is problematic. The standard p…

Bayesian neural networks fail at OOD detection, revealing fundamental issues.

We develop a convex analytic approach to analyze finite width two-layer ReLU networks. We first prove that an optimal solution to the regularized training problem can be characterized as extreme points of a convex set, where simple solutions are encouraged via its convex geometrical properties. We then leverage this ch…

The paper examines when NTK theory applies to real finite-width neural networks.

It is well-known that the distribution over functions induced through a zero-mean iid prior distribution over the parameters of a multi-layer perceptron (MLP) converges to a Gaussian process (GP), under mild conditions. We extend this result firstly to independent priors with general zero or non-zero means, and secondl…

In this paper, we shall use the Kähler geometry formulation to study the global behavior of the Ricci flow on $R^2$. The geometric feature of our Ricci flow is that it has finite width. Our aim is to determine the limiting metric (which corresponds an eternal Ricci flow) obtained by L.F.Wu. We can use the classificatio…

New framework connects two neural network theories, improving finite-width approximations.