Existing depth separation results for constant-depth networks essentially show that certain radial functions in , which can be easily approximated with depth networks, cannot be approximated by depth networks, even up to constant accuracy, unless their size is exponential in . However, the func…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Constant Depth”
The paper proves barriers to approximating functions with small weights and depth in neural networks.
problem Proving barriers to approximating functions with constant depth neural networks.
method Reduction to open problems and natural-proof barriers in circuit complexity, and a new approach to polynomially-bounded functions.
result There are fundamental barriers to proving results beyond depth 4 for constant-depth neural networks.
Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.
problem Approximating functions with depth 2 networks in high dimensions.
method Lower bound proof using worst-to-average-case random self-reducibility.
result Proves depth 2 networks can't approximate certain functions as well as depth 3 networks, resolving an open problem.
New algorithm learns random neural networks efficiently.
problem Learning random constant-depth neural networks efficiently.
method Presented a PTAS (Polynomial-Time Approximation Scheme) for learning random Xavier networks of fixed depth.
result For any fixed ε and depth i, there is a poly-time algorithm that learns random Xavier networks up to an additive error of ε.
Adversarial examples in deep ReLU networks with constant depth.
problem Adversarial perturbations lead to large output changes in deep ReLU networks.
method Analyzes the phenomenon in networks with independent Gaussian parameters and constant depth.
result Adversarial examples arise due to functions being close to linear.
Improved neural network depth-width trade-offs via dynamical systems.
problem Expressivity of neural networks in terms of depth and width.
method Connection with dynamical systems, focusing on periodic points and Lipschitz constants.
result Sharper width lower bounds for neural networks, yielding exponential depth-width separations.
CoT enhances transformer accuracy on serial tasks by enabling serial computation.
problem Improving accuracy of large language models on inherently serial problems.
method Integrating a chain of thought (CoT) into decoder-only transformers to enable serial computation.
result Constant-depth transformers with CoT can solve problems in AC^0, surpassing TC^0 without CoT.
This paper optimizes ReLU networks for approximating Hölder continuous functions.
problem Optimizing the approximation rate of ReLU networks in terms of width and depth.
method Constructive proof of ReLU networks' approximation power with specific width and depth constraints.
result Optimal approximation rate of ReLU networks with width and depth constraints.
Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activationsstat.ML
Smooth activations enable optimal error rates in neural networks for Sobolev function classes.
problem Achieving optimal approximation and estimation error rates for neural networks in Sobolev function classes.
method Study of neural networks with smooth activations, proving optimal rates via approximation and statistical properties.
result Constant-depth networks with smooth activations achieve optimal rates of approximation and estimation, demonstrating smoothness adaptivity.
A basic question in the theory of fault-tolerant quantum computation is to understand the fundamental resource costs for performing a universal logical set of gates on encoded qubits to arbitrary accuracy. Here we consider qubits encoded with constant space overhead (i.e. finite encoding rate) in the limit of arbitrari…
Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.
problem Challenges in choosing an appropriate learning rate for deep networks, especially as depth increases.
method Introduces Arithmetic-Mean P (AM-P), constraining network-wide average pre-activation second moment to a constant scale, combined with residual-aware He fan-in initialization.
result Demonstrates a scaling law for learning rates across depths, enabling zero-shot learning-rate transfer.
Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.
problem Understanding how size and depth of neural networks affect their ability to approximate benign functions.
method Analyzing ReLU networks for benign functions, proving barriers and explicit results.
result Explicit benign functions that cannot be approximated by networks of certain sizes or depths, showing barriers to size and depth separation.
Algorithm efficiently learns deep ReLU networks with polynomial runtime in depth and parameters.
problem Learning deep ReLU networks with polynomial runtime.
method Algorithm using filtered PCA and lattice polynomial analysis.
result First nontrivial results for networks of depth more than two with polynomial runtime.
Let be a closed essential surface in a hyperbolic 3-manifold with a toroidal cusp . The depth of in is the maximal distance from points of in to the boundary of . It will be shown that if is an essential pleated surface which is not coannular to the boundary torus of then the depth…
Constant Scalar Curvature Kähler Metrics on Resolutions of an Orbifold Singularity of Depth 1math.DG
New metrics found on orbifold resolutions with specific curvature.
problem Finding metrics with constant scalar curvature on orbifolds.
method Constructing metrics on resolutions of orbifolds with type I singularities.
result Constant scalar curvature Kähler metrics constructed on resolutions.
Study of deep linear neural networks with proportional width and depth.
problem Lack of descriptive power in Gaussian limit of deep linear neural networks.
method Proportional infinite-width infinite-depth limit for deep linear neural networks.
result Characterization of limiting distribution as a nontrivial mixture of Gaussians.
This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.
problem The Neural Tangent Kernel's behavior in overparameterized neural networks with large width and depth is unclear.
method Experimental and theoretical analysis of ReLU networks with large width and depth.
result The aggregate norm of hidden neuron deviations does not vanish in infinitely-wide ReLU networks, indicating non-trivial behavior.
The paper examines how the angle between inputs in ReLU networks decreases with depth, impacting training.
problem Depth degeneracy in neural networks, leading to constant function behavior on initialization.
method Combinatorial expansions and Monte Carlo experiments to analyze the angle between inputs in ReLU networks of increasing depth.
result The angle between inputs in ReLU networks decreases exponentially with depth, leading to constant function behavior on initialization.
Sharp representation theorems show depth benefits for ReLU networks.
problem Representing functions with ReLU networks of varying depths.
method Proved sharp representation results capturing depth benefits.
result Approximation rates improve with depth for less smooth functions.
We show that the standard stochastic gradient decent (SGD) algorithm is guaranteed to learn, in polynomial time, a function that is competitive with the best function in the conjugate kernel space of the network, as defined in Daniely, Frostig and Singer. The result holds for log-depth networks from a rich family of ar…
This paper introduces Schur-constant equilibrium distribution models of dimension n for arithmetic non-negative random variables. Such a model is defined through the (several orders) equilibrium distributions of a univariate survival function. First, the bivariate case is considered and analyzed in depth, stressing the…
Unified theory of deep neural networks with diverse activations.
problem Understanding the relationship between depth and complexity in deep neural networks.
method Developed a unified function space theory for deep networks with various activations.
result Unified theory provides meaningful complexity for deep networks with diverse activations.
Study fractal and regular geometry in deep neural networks.
problem Investigate geometric properties of neural networks.
method Analyze boundary volumes of excursion sets for different activations.
result Hausdorff dimension increases with depth for non-regular activations.
Derives token price process for AMM tokens, finds leverage effect and pricing discrepancies.
problem Derives token price process for AMM tokens.
method Derives CEV process for token price, derives closed-form option prices, introduces liquidity-adjusted Greeks.
result Token price process is CEV, with leverage effect and pricing discrepancies.
Randomly initialized ReLU networks of depth two can approximate smooth functions well.
problem Approximation power of two-layer networks of random ReLUs.
method Harmonic analysis and ridgelet representation theory for upper bounds, dimensionality arguments for lower bounds.
result Near-matching upper and lower bounds for -approximation and Sobolev norms.
Continuous-depth Evoformer reduces protein folding prediction time and resource usage.
problem Efficient protein structure prediction with reduced computational costs.
method Continuous-depth formulation of Evoformer using Neural Ordinary Differential Equations (Neural ODEs).
result The continuous-time Evoformer achieves constant memory cost and improved efficiency.
The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationstat.ML
ResNets approximate log-Gaussian at initialization, improving network performance.
problem Understanding the initialization behavior of deep neural networks like ResNets.
method Analyzing ReLU ResNets in the infinite-depth-and-width limit, showing log-Gaussian behavior.
result ResNets at initialization exhibit hypoactivation and interlayer correlations, which are not captured by Gaussian limits.
We show that there is a simple (approximately radial) function on , expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all kn…
We address representational challenges in normalizing flows, particularly depth and conditioning issues.
problem Challenges in training normalizing flows, including vanishing/exploding gradients and poor conditioning.
method Analyzes representational aspects of depth and conditioning in normalizing flows, proving theoretical bounds and investigating phenomena.
result Proves that shallow affine coupling networks are universal approximators in Wasserstein distance if ill-conditioning is allowed.
Transformers can learn Markov processes with constant depth, surprising results.
problem Understanding how transformers learn context in Markov processes.
method Empirical study and theoretical analysis of attention-based transformers on Markov data.
result Transformers with constant depth can achieve low test loss on Markov sequences, matching empirical and theoretical findings.
Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory. In a seminal paper, Telgarsky highlig…
This paper improves neural network approximation for analytic functions with adjustable depth and width.
problem Approximating analytic functions using neural networks with depth and width parameters.
method Characterizes approximation rates as a joint function of width (N) and depth (L) for ReLU networks.
result Establishes upper bounds for analytic function approximation rates of O(N^(-CL^τ)) with τ influenced by N and L.
This paper studies bounds for the Lipschitz constant of random neural networks.
problem Quantifying the worst-case robustness of neural networks against adversarial perturbations.
method Analyzes upper and lower bounds for the Lipschitz constant of random ReLU neural networks under specific initialization conditions.
result For deep networks, the upper bound is larger than the lower bound by a logarithmic factor in width.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constan…
This paper extends depth separation results to piece-wise oscillatory functions.
problem Approximating functions with piece-wise oscillatory structure using neural networks.
method Extends existing results to piece-wise oscillatory functions using proof strategy from (Eldan and Shamir, 2016).
result Approximation by one-hidden-layer networks holds at a poly(d) rate for functions with constant domain radius and oscillation rate.
Let M be a geometrically finite pinched negatively curved Riemannian manifold with at least one cusp. Inspired by the theory of diophantine approximation of a real (or complex) number by rational ones, we develop a theory of approximation of geodesic lines starting from a given cusp by ones returning to it. We define a…
This paper shows that scientific discovery can be efficiently learned via compositional function trees, reducing the sample complexity.
problem Statistical and computational intractability of scientific discovery via symbolic regression.
method PAC learning approach focusing on compositional function trees built from a finite vocabulary of smooth operators.
result The Rademacher complexity and excess risk are controlled by depth and Lipschitz constants of the base operators, leading to finite-union bounds and high-probability risk bounds.
Deep neural networks (DNNs) have emerged as a popular mathematical tool for function approximation due to their capability of modelling highly nonlinear functions. Their applications range from image classification and natural language processing to learning-based control. Despite their empirical successes, there is st…
In this paper, we present a multi-period trading model in the style of Kyle (1985)'s inside trading model, by assuming that there are at least two insiders in the market with long-lived private information, under the requirement that each insider publicly discloses his stock trades after the fact. Based on this model, …
New framework improves robustness of implicit neural networks.
problem Ill-posedness and convergence instability in implicit neural networks.
method NEMON framework based on contraction theory for norm, including well-posedness condition, average iteration, and input-output Lipschitz constant regularization.
result Improved accuracy and robustness of implicit models with smaller input-output Lipschitz bounds.
This paper analyzes the Lipschitz constants of deep neural networks with random weights.
problem Estimating the Lipschitz constants of deep neural networks with random parameters.
method High probability upper and lower bounds derived for ReLU neural networks with He initialization.
result The behavior of the Lipschitz constant varies significantly between and .
Uniform scaling limits in AdamW-trained transformers converge to ODEs.
problem Understanding the dynamics of large-depth transformers trained with AdamW.
method Modeling transformer dynamics as an interacting particle system coupled through attention, proving convergence to ODEs.
result The joint dynamics of hidden states and backpropagated variables converge uniformly to an ODE system.
A new depth measure for non-convex data supports, faster than halfspace depth.
problem Non-convex data supports in multivariate statistics.
method Extending halfspace depth to Reproducing Kernel Hilbert Space (RKHS).
result The new depth measure is consistent and can be computed faster.
The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.
problem The challenge of approximating Tukey's depth in high dimensions.
method The study examines randomized algorithms for approximating Tukey's depth for log-concave isotropic data.
result Randomized algorithms correctly approximate maximal depth and close to zero depths but not intermediate depths.
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.
This paper connects functional data analysis with machine learning techniques.
problem Lack of theoretical analysis for functional depths.
method Viewing functional depths as kernel mean embeddings in machine learning.
result Facilitates answers to open questions about functional depths.
Deep linear networks minimize sharpness, avoiding large eigenvalues.
problem Understanding optimization dynamics in deep linear networks for regression.
method Analyzing sharpness (largest eigenvalue of Hessian) of minimizers and gradient flow solutions.
result Gradient flow implicitly regularizes towards flat minima, with sharpness bounded by a constant.