Complex-valued neural networks can approximate any continuous function with bounded widths and depths.
problem Approximating continuous functions with complex-valued neural networks of bounded widths and depths.
method Analyzing activation functions and proving universality for complex-valued networks.
result Deep narrow complex-valued networks are universal if and only if their activation function is neither holomorphic, nor antiholomorphic, nor R-affine. The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let n be the number of inputs neurons, m be the number of output neurons, and let ρ be any nonaff…
Following the seminal idea of Tukey, data depth is a function that measures how close an arbitrary point of the space is located to an implicitly defined center of a data cloud. Having undergone theoretical and computational developments, it is now employed in numerous applications with classification being the most po…
Uniform consistency proven for spatial distribution and depth estimators in any dimension.
problem Uniform consistency of spatial distribution and depth estimators in arbitrary dimensions.
method Proof of uniform L1-consistency using sample size n as the only dependency. result Consistency rate is independent of dimension d and sample size n. This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed din≥1, what is the minimal width w so that neural nets with ReLU activations, input dimension din, hidden layer widths at most w, and …
Over-parameterized CNNs show U-shaped test risk with depth increase.
problem Understanding the impact of depth on test risk in over-parameterized CNNs.
method Empirical image classification experiments and linear regression framework.
result Test risk is U-shaped with increasing depth in over-parameterized CNNs.
The paper proves neural network identifiability for a broad range of nonlinearities.
problem Can a neural network's architecture, weights, and biases be uniquely determined by its input-output map?
method Derive necessary genericity conditions for identifiability of neural networks of arbitrary depth and connectivity with an arbitrary nonlinearity.
result Construct a family of nonlinearities for which these genericity conditions are minimal, necessary, and sufficient.
Proves necessity of at least log2(n) layers to compute maximum of n numbers.
problem Computing the maximum of n numbers with ReLU neural networks.
method Uses lattice polytopes and duality with Newton polytopes to prove depth lower bounds.
result Proves that log2(n) hidden layers are necessary and sufficient.
A new depth measure and median defined on Hadamard manifolds.
problem Statistical depth and median on Hadamard manifolds.
method Horospherical depth and Busemann median defined using renormalized distance functions.
result The Busemann median exists for every Borel probability measure on Hadamard manifolds.
Deep architecture such as hierarchical semi-Markov models is an important class of models for nested sequential data. Current exact inference schemes either cost cubic time in sequence length, or exponential time in model depth. These costs are prohibitive for large-scale problems with arbitrary length and depth. In th…
Monotone neural networks can approximate and interpolate functions efficiently.
problem Understanding the efficiency and expressiveness of monotone neural networks.
method Solving the monotone interpolation problem using depth-4 networks and comparing size bounds with arbitrary networks.
result Monotone neural networks can approximate and interpolate functions efficiently, but may require exponential size in high dimensions.
UDN adapts depth to data complexity, outperforming standard neural networks.
problem Adapting neural network depth to data complexity.
method Variational inference for infinitely deep neural networks with a novel algorithm.
result UDN outperforms standard neural networks and other infinite-depth approaches.
Recent advances in bandit tools and techniques for sequential learning are steadily enabling new applications and are promising the resolution of a range of challenging related problems. We study the game tree search problem, where the goal is to quickly identify the optimal move in a given game tree by sequentially sa…
David Gabai showed that disk decomposable knot and link complements carry taut foliations of depth one. In an arbitrary sutured 3-manifold M, such foliations F, if they exist at all, are determined up to isotopy by an associated ray [F] issuing from the origin in H^1(M;R) and meeting points of the integer lattice H^1(M…
Study shows depth improves trainability of neural networks by improving kernel conditioning.
problem Improving trainability of neural networks with random initialization and overparameterization.
method Analyzes the role of depth in training neural networks, proving that depth improves conditioning of kernel matrices.
result General result showing depth improves trainability of neural networks by improving the conditioning of kernel matrices.
A new depth measure captures anomalies in time series data.
problem Anomaly detection in time series data.
method Convex hull area of sampled curves for functional depth.
result Captures gradual departures from centrality.
Deep ReLU networks can approximate smooth functions nearly optimally.
problem Approximating smooth functions with deep neural networks.
method Using Taylor expansions and deep ReLU network approximations, the paper establishes optimal approximation error bounds.
result Deep ReLU networks of width and depth O(NlnN) and O(LlnL) can approximate f∈Cs([0,1]d) with an error O(∥f∥Cs([0,1]d)N−2s/dL−2s/d). This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width wmin(d) so that ReLU nets of width wmin(d) (and arbitrary depth) can approximate any continuous functio…
Sharp representation theorems show depth benefits for ReLU networks.
problem Representing functions with ReLU networks of varying depths.
method Proved sharp representation results capturing depth benefits.
result Approximation rates improve with depth for less smooth functions.
In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are gu…
Unified theory of deep neural networks with diverse activations.
problem Understanding the relationship between depth and complexity in deep neural networks.
method Developed a unified function space theory for deep networks with various activations.
result Unified theory provides meaningful complexity for deep networks with diverse activations.
The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities and pooling layers, and to networks with different architectures such as feed forw…
Bayesian inference with deep, weakly nonlinear networks is solved rigorously.
problem Bayesian inference with neural networks of specific structure.
method Perturbative analysis of fully connected neural networks with a shaped nonlinearity.
result Neural network Bayesian inference can be equivalent to kernel methods under certain conditions.
This paper optimizes ReLU networks for approximating Hölder continuous functions.
problem Optimizing the approximation rate of ReLU networks in terms of width and depth.
method Constructive proof of ReLU networks' approximation power with specific width and depth constraints.
result Optimal approximation rate of ReLU networks with width and depth constraints.
Hardness proven for neural networks with natural weights.
problem Difficulty in learning neural networks with weights from natural distributions.
method Proved hardness for depth-2 networks with natural weights distributions.
result Most networks are hard to learn with natural weights.
Deep and wide networks are shown to be equivalent in terms of their capability.
problem The relationship between the width and depth of neural networks.
method Formulated transforms to map networks, used polynomial representations.
result Deep and wide networks are quasi-equivalent with an arbitrarily small error.
Study CR manifolds focusing on Levi and contact-nondegeneracy.
problem Investigate CR algebras and their properties.
method Locally homogeneous case analysis, cross-marked painted root diagrams.
result Establish correspondences with CR-algebra properties and combinatorics.
New activation functions achieve arbitrary-accuracy Sobolev approximation by fixed-size neural networks.
problem Approximation of Sobolev functions by neural networks
method Elementary Universal Activation Function and Differentiable Universal Activation Functions
result Arbitrary-accuracy Sobolev approximation by fixed-size neural networks
This work generalizes bounds on the number of linear regions in CPWL NNs.
problem Determining the number of linear regions in CPWL neural networks is challenging.
method Generalized bounds on the maximal number of linear regions for arbitrary CPWL activation functions.
result Depth significantly increases the number of linear regions, but not exponentially.
We obtain new invariants of topological link concordance and homology cobordism of 3-manifolds from Hirzebruch-type intersection form defects of towers of iterated p-covers. Our invariants can extract geometric information from an arbitrary depth of the derived series of the fundamental group, and can detect torsion wh…
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.
FUSE neural centrality framework improves data point measurement in high dimensions.
problem Measuring centrality in high-dimensional data is expensive and unstable.
method Combines global and local heads trained on arbitrary representations.
result Reveals meaningful classical ordering and competitive performance.
Cohomology fractals are visual representations of cohomology classes on hyperbolic 3-manifolds.
problem Visualizing cohomology classes on hyperbolic 3-manifolds.
method Cohomology fractals are images associated to cohomology classes. They are related to limit sets of Kleinian groups but differ in key aspects. An implementation using ideal triangulations and ray-casting is presented.
result Cohomology fractals allow for real-time zooming in any direction at arbitrary depth.
We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Montúfar and Ay,…
We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to a well known Bayesian model. This interpretation might offer an explanation to some of dropout's key properties, such as its robustness to over-fitt…
New network approximates functions with error decreasing with network width and depth.
problem Approximating functions with high-dimensional data.
method Floor-ReLU networks with specific width and depth.
result Approximation error decreases as network width and depth increase.
A new depth measure for non-convex data supports, faster than halfspace depth.
problem Non-convex data supports in multivariate statistics.
method Extending halfspace depth to Reproducing Kernel Hilbert Space (RKHS).
result The new depth measure is consistent and can be computed faster.
The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.
problem The challenge of approximating Tukey's depth in high dimensions.
method The study examines randomized algorithms for approximating Tukey's depth for log-concave isotropic data.
result Randomized algorithms correctly approximate maximal depth and close to zero depths but not intermediate depths.
A basic question in the theory of fault-tolerant quantum computation is to understand the fundamental resource costs for performing a universal logical set of gates on encoded qubits to arbitrary accuracy. Here we consider qubits encoded with constant space overhead (i.e. finite encoding rate) in the limit of arbitrari…
This paper connects functional data analysis with machine learning techniques.
problem Lack of theoretical analysis for functional depths.
method Viewing functional depths as kernel mean embeddings in machine learning.
result Facilitates answers to open questions about functional depths.
We introduce Bayesian multi-tensor factorization, a model that is the first Bayesian formulation for joint factorization of multiple matrices and tensors. The research problem generalizes the joint matrix-tensor factorization problem to arbitrary sets of tensors of any depth, including matrices, can be interpreted as u…
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
A new depth measure based on optimal control theory captures multi-modal data.
problem Statistical depths for high-dimensional data.
method Eikonal equations and optimal control theory.
result The new depth measure is robust under adversarial models.
New method uses higher-order Langevin dynamics for efficient parallel sampling.
problem Efficient parallel sampling from high-dimensional log-concave distributions.
method Combines higher-order Langevin dynamics with blockwise Lagrange polynomial interpolation.
result Reduces the number of parallel points required for a target accuracy.
Paper analyzes soft tree ensembles using NTK, finding only leaf count matters.
problem Understanding impact of various tree architectures in ensemble learning.
method Formulated and analyzed Neural Tangent Kernel (NTK) for soft tree ensembles.
result Only the number of leaves at each depth is relevant for tree architecture in ensemble learning.
Transforms uniform learners to work under arbitrary distributions efficiently.
problem Learning under arbitrary distributions from uniform learners.
method Black-box transformation using decision tree decomposition.
result Efficient transformation with runtime scaling with distribution complexity.
New approach uses loss functions to extend data depth for anomaly detection.
problem Anomaly detection in high-dimensional data.
method Introducing loss depths to generalize halfspace depth.
result New loss depths improve anomaly detection efficiency and interpretability.
Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.
problem Approximating functions with depth 2 networks in high dimensions.
method Lower bound proof using worst-to-average-case random self-reducibility.
result Proves depth 2 networks can't approximate certain functions as well as depth 3 networks, resolving an open problem.