arXiv research

Bayesian neural networks learn efficiently at infinite width, matching polynomial-width performance.

Study shows polynomial-width neural networks can closely approximate infinite-width networks in polynomial time.

Polynomial neural networks explore thresholds for maximum expressiveness.

Wide networks with polynomial activations have proven asymptotic behavior.

New findings on depth vs. width in neural networks, showing depth can improve learnability.

Study on identifiability of deep polynomial neural networks.

Study shows neural networks trained with GD converge to Gaussian processes with polynomial decay.

This paper explores BDL hyperparameters for robust polynomial mapping with noise.

We give a general fixed parameter tractable algorithm to compute quantum invariants of links presented by diagrams, whose complexity is singly exponential in the carving-width (or the tree-width) of the diagram. In particular, we get a $O(N^{\frac{3}{2} \mathrm{cw}} \mathrm{poly}(n))$ time algorithm to compute any Resh…

Khovanov homology is a bigraded Z-module that categorifies the Jones polynomial. The support of Khovanov homology lies on a finite number of slope two lines with respect to the bigrading. The Khovanov width is essentially the largest horizontal distance between two such lines. We show that it is possible to generate in…

We study deep neural networks with polynomial activations, particularly their expressive power. For a fixed architecture and activation degree, a polynomial neural network defines an algebraic map from weights to polynomials. The image of this map is the functional space associated to the network, and it is an irreduci…

We show that the zeroes of the Alexander polynomial of a Lorenz knot all lie in some annulus whose width depends explicitly on the genus and the braid index of the considered knot.

Shallow neural networks can represent polynomials efficiently.

We show that the class of strongly connected graphical models with treewidth at most k can be properly efficiently PAC-learnt with respect to the Kullback-Leibler Divergence. Previous approaches to this problem, such as those of Chow ([1]), and Ho gen ([7]) have shown that this class is PAC-learnable by reducing it to …

New findings show depth is more important than width in neural networks.

New insights into $p$-widths of surfaces, proving optimality and calculating constants.

Paper studies ResNet dynamics using NTH, reducing width requirement.

Shallow nonlinear networks can separate classes linearly with polynomially scaling width.

Recent theoretical work has guaranteed that overparameterized networks trained by gradient descent achieve arbitrarily low training error, and sometimes even low test error. The required width, however, is always polynomial in at least one of the sample size $n$, the (inverse) target error $1/ε$, and the (inverse) fail…

This paper studies the expressive power of graph neural networks falling within the message-passing framework (GNNmp). Two results are presented. First, GNNmp are shown to be Turing universal under sufficient conditions on their depth, width, node attributes, and layer expressiveness. Second, it is discovered that GNNm…

Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with $N$ ($N$ being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width $N$ following the inpu…

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N…

We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation (Bölcskei et al., 2018) of polynomials, windowed sinusoidal functions, one-dimensional oscillatory textures, and the Weierstrass function, a fractal function which is continuous but nowhere differentiable. Together with thei…

The study reveals a transition in neural network performance from infinite-width to variance-limited behavior as dataset size increases.

Two new algorithms improve robust PCA and Schatten packing.

New algorithm speeds up knot polynomial calculations.

Deep neural networks with piecewise-polynomial activations can approximate smooth functions and their derivatives.

The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let $n$ be the number of inputs neurons, $m$ be the number of output neurons, and let $ρ$ be any nonaff…

Deep networks can perfectly classify two low-dimensional manifolds on a sphere with large depth and width.

Deep networks can be biased to learn top eigenfunctions of the kernel outside the training set.

Adversarial examples in deep ReLU networks with constant depth.

This paper presents a new anytime algorithm for the marginal MAP problem in graphical models. The algorithm is described in detail, its complexity and convergence rate are studied, and relations to previous theoretical results for the problem are discussed. It is shown that the algorithm runs in polynomial-time if the …

The paper proves deep neural networks with analytic activation can approximate any function.

Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works has been focusing on training neural networks with one hidden layer. The theory of multi-layer networks remains largely …

This paper improves neural network approximation for analytic functions with adjustable depth and width.

Different variants of MFDFA technique are applied in order to investigate various (artificial and real-world) time series. Our analysis shows that the calculated singularity spectra are very sensitive to the order of the detrending polynomial used within the MFDFA method. The relation between the width of the multifrac…

The paper provides non-asymptotic Edgeworth expansions for neural network outputs.

A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size $n$ and the inverse of the target error $ε^{-1}$, deep neural networks learned by (stochastic) gradient descent enjoy …

Gradient methods improve deep network training with tighter bounds and faster convergence.

While classic studies proved that wide networks allow universal approximation, recent research and successes of deep learning demonstrate the power of deep networks. Based on a symmetric consideration, we investigate if the design of artificial neural networks should have a directional preference, and what the mechanis…

Paper proves convergence rates for Gaussian kernel ridge regression.

Study robustness of polynomial neural networks using algebraic geometry.

A very simple $\mathbb R^3$ realization of the Möbius strip, significantly simpler than the common one, is given. For any, however large width/length ratio of the strip, it is shown that this realization, in contrast with the common one, is the union of a vertical segment and the graph of a simple rational function on …

We prove that for an $L$-layer fully-connected linear neural network, if the width of every hidden layer is $\tildeΩ(L \cdot r \cdot d_{\mathrm{out}} \cdot κ^3 )$, where $r$ and $κ$ are the rank and the condition number of the input data, and $d_{\mathrm{out}}$ is the output dimension, then gradient descent with Gaussi…

Two-layer NN with channel attention learns low-degree spherical polynomials efficiently.

This work finds a point with small test error in polynomial time for mildly overparameterized neural nets.

The study analyzes transfer learning in infinite-width neural networks, improving generalization on target tasks.

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network functio…