Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.
problem Approximating functions with depth 2 networks in high dimensions.
method Lower bound proof using worst-to-average-case random self-reducibility.
result Proves depth 2 networks can't approximate certain functions as well as depth 3 networks, resolving an open problem.
Proves necessity of at least log2(n) layers to compute maximum of n numbers.
problem Computing the maximum of n numbers with ReLU neural networks.
method Uses lattice polytopes and duality with Newton polytopes to prove depth lower bounds.
result Proves that log2(n) hidden layers are necessary and sufficient.
Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.
problem Understanding how size and depth of neural networks affect their ability to approximate benign functions.
method Analyzing ReLU networks for benign functions, proving barriers and explicit results.
result Explicit benign functions that cannot be approximated by networks of certain sizes or depths, showing barriers to size and depth separation.
Improved neural network depth-width trade-offs via dynamical systems.
problem Expressivity of neural networks in terms of depth and width.
method Connection with dynamical systems, focusing on periodic points and Lipschitz constants.
result Sharper width lower bounds for neural networks, yielding exponential depth-width separations.
Proves SQ lower bounds for learning two-hidden-layer neural networks.
problem Learning two-hidden-layer ReLU networks with Gaussian inputs.
method Refined lifting procedure to reduce Boolean PAC learning to Gaussian.
result Superpolynomial SQ lower bounds for Gaussian inputs.
Study shows depth improves generalization in deep learning models.
problem Understanding why and when depth improves generalization in deep learning.
method Implementation-agnostic state-transition model to analyze depth and generalization.
result Identifies geometric and semigroup mechanisms that keep entropy contribution saturated or polynomial, clarifying depth's statistical advantage.
Transformers can learn noisy linear systems with depth and IID data.
problem Learning noisy linear dynamical systems with transformers.
method Theoretical analysis of multi-layer and single-layer transformers with respect to L2-testing loss. result Single-layer transformers have a non-diminishing lower bound on approximation error, suggesting depth separation.
Probabilistic bounds on neuron death in deep networks, showing depth can be increased indefinitely.
problem Understanding neuron death in deep rectifier networks and its impact on model trainability.
method Deriving upper and lower bounds on neuron death probability as a function of model hyperparameters.
result The probability of neuron death decreases as network depth increases, provided width increases proportionally.
The paper improves bounds on skein tree depth and delta-crossing numbers for knots and links.
problem Improving bounds on skein tree depth and delta-crossing numbers for knots and links.
method Theoretical and computational analysis of skein trees and knot invariants.
result New upper and lower bounds on skein tree depth and delta-crossing numbers are derived.
Randomly initialized ReLU networks of depth two can approximate smooth functions well.
problem Approximation power of two-layer networks of random ReLUs.
method Harmonic analysis and ridgelet representation theory for upper bounds, dimensionality arguments for lower bounds.
result Near-matching upper and lower bounds for L2-approximation and Sobolev norms. Study shows shorter words for group elements in surface groups and RAAGs.
problem Understanding the structure of surface groups and RAAGs through word lengths.
method Proved lower bounds on shortest words representing nontrivial elements in their lower central series.
result Found that the shortest words are shorter for surface groups and RAAGs.
The paper improves bounds on the complexity of computing link polynomials.
problem Computing link polynomials by the skein relation is complex.
method Proved new upper and lower bounds on skein tree depth.
result New bounds on skein tree depth are stronger than previous ones.
Revisits Gaussian process model with spherical harmonics for scalable deep learning.
problem Scaling Gaussian process models to large input dimensions with high frequency learning.
method Introduces new kernels related to deep models, variational learning of spherical harmonic phases, and sparseness in eigenbasis.
result Enables scaling to larger input dimensions and learning of high frequency variations.
We address representational challenges in normalizing flows, particularly depth and conditioning issues.
problem Challenges in training normalizing flows, including vanishing/exploding gradients and poor conditioning.
method Analyzes representational aspects of depth and conditioning in normalizing flows, proving theoretical bounds and investigating phenomena.
result Proves that shallow affine coupling networks are universal approximators in Wasserstein distance if ill-conditioning is allowed.
Deep forests enhance expressiveness exponentially with depth, not width or tree size.
problem Understanding the role of depth, width, and tree size in deep forest performance.
method Provided upper and lower bounds on deep forest approximation complexity.
result Depth exponentially enhances deep forest expressiveness.
New findings show depth is more important than width in neural networks.
problem Understanding the role of width and depth in neural networks.
method Constructed networks with bounded weights and width at most d+2, showing depth plays a more significant role.
result Depth is more important than width in the expressive power of neural networks.
This work gives an in-depth derivation of the trainable evidence lower bound obtained from the marginal joint log-Likelihood with the goal of training a Multi-Modal Variational Autoencoder (M2VAE).
It has recently been shown that for compressive sensing, significantly fewer measurements may be required if the sparsity assumption is replaced by the assumption the unknown vector lies near the range of a suitably-chosen generative model. In particular, in (Bora {\em et al.}, 2017) it was shown roughly O(klogL) r…
This paper studies the expressive power of graph neural networks falling within the message-passing framework (GNNmp). Two results are presented. First, GNNmp are shown to be Turing universal under sufficient conditions on their depth, width, node attributes, and layer expressiveness. Second, it is discovered that GNNm…
The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes o…
New research limits how deep neural networks can be for certain functions.
problem Understanding the depth required for neural networks to represent specific functions.
method Mixed-integer optimization, polyhedral theory, tropical geometry.
result Neural networks with more than one layer are necessary to represent certain functions.
Existing Rademacher complexity bounds for neural networks rely only on norm control of the weight matrices and depend exponentially on depth via a product of the matrix norms. Lower bounds show that this exponential dependence on depth is unavoidable when no additional properties of the training data are considered. We…
Motivated by the resurgence of neural networks in being able to solve complex learning tasks we undertake a study of high depth networks using ReLU gates which implement the function x↦max{0,x}. We try to understand the role of depth in such neural networks by showing size lowerbounds against such network …
This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected traject…
This work generalizes bounds on the number of linear regions in CPWL NNs.
problem Determining the number of linear regions in CPWL neural networks is challenging.
method Generalized bounds on the maximal number of linear regions for arbitrary CPWL activation functions.
result Depth significantly increases the number of linear regions, but not exponentially.
Looped Transformers improve robustness and expressivity in in-context learning for diverse tasks.
problem Improving robustness and expressivity in in-context learning for diverse tasks.
method Study in-context linear regression with diverse tasks, focusing on depth and looping.
result Looped Transformers exhibit similar expressive power and are provably robust under mild assumptions.
This paper studies bounds for the Lipschitz constant of random neural networks.
problem Quantifying the worst-case robustness of neural networks against adversarial perturbations.
method Analyzes upper and lower bounds for the Lipschitz constant of random ReLU neural networks under specific initialization conditions.
result For deep networks, the upper bound is larger than the lower bound by a logarithmic factor in width.
A new statistical concept, lepto-variance, is defined for stock returns using Regression Trees.
problem Understanding the underlying structure of stock returns using statistical methods.
method Defining lepto-variance as the variance that cannot be removed by any regression tree of a specific depth and analyzing stock returns with 1- and 2-bit Regression Trees.
result Lepto-variance quantifies the resolving power of Regression Trees for stock returns, decomposing total variance into lepto-variance and macro-variance.
Defines slice depth for 2-knots and sets upper bounds for specific knots.
problem Determining the minimum dimension for a 2-knot to be slice.
method Introduces slice depth, defines it for 2-knots, and provides upper bounds for specific knot types.
result Upper bounds for slice depth of certain 2-knots.
The paper analyzes how gradient descent implicitly regularizes solutions in overparameterized neural networks, revealing depth-dependent regularization effects.
problem Understanding implicit regularization in overparameterized linear neural networks for regression problems.
method Analyzing the approximation error between gradient flow limit points and ℓ1-minimization solutions, deriving tight upper and lower bounds. result The approximation error decreases linearly for D≥3 and at a slower rate for D=2, linked to null space property constants. Sharp representation theorems show depth benefits for ReLU networks.
problem Representing functions with ReLU networks of varying depths.
method Proved sharp representation results capturing depth benefits.
result Approximation rates improve with depth for less smooth functions.
Robust estimation under Huber's ε-contamination model has become an important topic in statistics and theoretical computer science. Statistically optimal procedures such as Tukey's median and other estimators based on depth functions are impractical because of their computational intractability. In this paper, we est…
Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.
problem Expressing high-dimensional Lipschitz functions with shallow ReLU networks.
method Established lower bounds on shallow network complexity for polynomial approximation.
result Shallow ReLU networks suffer from the curse of dimensionality for Lipschitz functions.
The paper proves barriers to approximating functions with small weights and depth in neural networks.
problem Proving barriers to approximating functions with constant depth neural networks.
method Reduction to open problems and natural-proof barriers in circuit complexity, and a new approach to polynomially-bounded functions.
result There are fundamental barriers to proving results beyond depth 4 for constant-depth neural networks.
Paper proves depth bounds for taut foliations using instanton Floer homology.
problem Finding depth bounds for taut foliations in sutured manifolds.
method Sutured instanton Floer homology, adapted to monopole and Heegaard Floer settings.
result Dimension of sutured instanton Floer homology bounds minimal depth of taut foliations.
Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory. In a seminal paper, Telgarsky highlig…
Deep linear networks minimize sharpness, avoiding large eigenvalues.
problem Understanding optimization dynamics in deep linear networks for regression.
method Analyzing sharpness (largest eigenvalue of Hessian) of minimizers and gradient flow solutions.
result Gradient flow implicitly regularizes towards flat minima, with sharpness bounded by a constant.
Data depth aids in identifying anomalies in multivariate data.
problem Detecting abnormal observations in multivariate datasets.
method Using data depth to assign abnormality labels to observations with lower depth values.
result Data depth effectively identifies anomalies in multivariate settings.
We show that deep narrow Boltzmann machines are universal approximators of probability distributions on the activities of their visible units, provided they have sufficiently many hidden layers, each containing the same number of units as the visible layer. We show that, within certain parameter domains, deep Boltzmann…
Neural operators improve solving Helmholtz equation for various wave speeds.
problem Neural operators struggle with out-of-distribution scenarios for high-frequency waves.
method Proposed a subfamily of neural operators with stochastic depth for enhanced approximation of the Helmholtz equation.
result Neural operators with stochastic depth outperform standard models in out-of-distribution scenarios.
DiffEnc improves diffusion models by adding flexibility and achieving better likelihood on CIFAR-10.
problem Improving the likelihood of diffusion models on image datasets.
method Introducing a data- and depth-dependent mean function and a free weight parameter for noise variance.
result Achieved statistically significant improvement in likelihood on CIFAR-10.
Unified theory of deep neural networks with diverse activations.
problem Understanding the relationship between depth and complexity in deep neural networks.
method Developed a unified function space theory for deep networks with various activations.
result Unified theory provides meaningful complexity for deep networks with diverse activations.
This paper extends depth separation results to piece-wise oscillatory functions.
problem Approximating functions with piece-wise oscillatory structure using neural networks.
method Extends existing results to piece-wise oscillatory functions using proof strategy from (Eldan and Shamir, 2016).
result Approximation by one-hidden-layer networks holds at a poly(d) rate for functions with constant domain radius and oscillation rate.
A new depth measure for non-convex data supports, faster than halfspace depth.
problem Non-convex data supports in multivariate statistics.
method Extending halfspace depth to Reproducing Kernel Hilbert Space (RKHS).
result The new depth measure is consistent and can be computed faster.
Paper uses statistical depth to create DP estimators for regression.
problem Creating differentially private estimators in high dimensions.
method Uses halfspace and regression depth to analyze maximum influence and construct DP estimators.
result New DP estimators for location and regression show favorable performance.
High-dimensional kernel regression struggles due to rotational invariance.
problem Kernel ridge regression struggles in high dimensions due to rotational invariance.
method Analysis of kernel properties and their impact on high-dimensional data.
result Lower bound on generalization error for high-dimensional kernel regression.
Our study analyzes how neural network initialization affects privacy and utility in overparameterized models.
problem Privacy and utility trade-off in overparameterized neural networks.
method Analytical proof of KL divergence privacy bound, focusing on initialization, width, and depth.
result Privacy bound improvement with increasing depth under certain initializations, degradation under others.
An efficient algorithm for aligning diffusion trees to networks with information asymmetry.
problem Aligning diffusion trees to networks with information asymmetry.
method Tree correlation tests for extracting alignment information.
result Explicit lower bounds on the probability of correct matches for each vertex on the diffusion tree.