arXiv research

Empirical study compares wide neural networks to kernel methods, resolving open questions.

Study examines dependence properties of Bayesian neural network units in finite-width networks.

NTK theory fails to predict practical behavior of large-width neural networks.

Unified spectral framework for μP under joint width-depth scaling.

Wide CNNs outperform infinite width networks, revealing scaling laws.

Theoretical limits of deep residual networks show consistent covariance structures.

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

Deep forests enhance expressiveness exponentially with depth, not width or tree size.

Wide neural networks can benefit from multi-task learning in their infinite-width limit.

Wide neural networks can degrade performance, contrary to conventional wisdom.

Empirical study shows standard CNNs deviate from NTK predictions.

Study deep maxout networks and their equivalence to Gaussian processes.

Stochastic neural networks with infinite width become deterministic, reducing training variance.

The paper improves theoretical bounds on deep neural networks' convergence.

Linearized attention fails to converge to NTK limit even at large widths.

New method combines HQR and WACI for better time series prediction intervals.

Novel algorithms improve efficiency of finite width NTK computation.

The paper addresses the gap between theoretical and practical confidence set widths in universal inference.

The paper extends infinite-width analysis to neural network Jacobians, revealing convergence to Gaussian processes and linear ODEs.

Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.

New optimizers control network width scaling, improving stability and transfer across different model sizes.

This paper explores how neural network width and depth behave as they approach infinity.

New analysis shows optimal embedding learning rate depends on vocabulary size, not just model width.

New method trains shallow neural networks with subquadratic width scaling.

Bayesian Bits unifies quantization and pruning through gradient optimization.

Joslim optimizes both width and weight configurations for slimmable neural networks, improving model efficiency.

We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.

The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let $n$ be the number of inputs neurons, $m$ be the number of output neurons, and let $ρ$ be any nonaff…

Our work proves convergence to low robust training loss for polynomial width ReLU networks.

This work establishes the equivalence between neural networks and support vector machines.

Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…

A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size $n$ and the inverse of the target error $ε^{-1}$, deep neural networks learned by (stochastic) gradient descent enjoy …

The book explains deep learning theory and how networks learn nontrivial representations.

We define the Wirtinger width of a knot. Then we prove the Wirtinger width of a knot equals its Gabai width. The algorithmic nature of the Wirtinger width leads to an efficient technique for establishing upper bounds on Gabai width. As an application, we use this technique to calculate the Gabai width of approximately …

This paper speeds up K-FAC for deep learning by focusing on only a few eigen-modes.

Empirical study compares finite- and infinite-width BNNs, revealing performance differences under model mismatch.

This paper explores BDL hyperparameters for robust polynomial mapping with noise.

Better uncertainty estimates for neural networks using Gaussian process priors.

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

The isospectral problem for p-widths is solved using Zoll metrics on S^2.

Width trees link link invariants and bridge number.

Batch normalization makes deep neural networks' representations increasingly orthogonal.

Graph homomorphism numbers embed graphs for classification.

Lectures on deep learning properties in infinite and large-width networks.

Computed p-widths for hemisphere, first for manifolds with boundary.

Polygon $p$-widths are found via billiard trajectories.

A number of results for C$^2$-smooth surfaces of constant width in Euclidean 3-space ${\mathbb{E}}^3$ are obtained. In particular, an integral inequality for constant width surfaces is established. This is used to prove that the ratio of volume to cubed width of a constant width surface is reduced by shrinking it along…

New framework explains fast transfer of hyperparameters across model scales.