arXiv research

The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

Deep forests enhance expressiveness exponentially with depth, not width or tree size.

Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.

We survey results on neural network expressivity described in "On the Expressive Power of Deep Neural Networks". The paper motivates and develops three natural measures of expressiveness, which all display an exponential dependence on the depth of the network. In fact, all of these measures are related to a fourth quan…

A new depth measure for non-convex data supports, faster than halfspace depth.

Quantum algorithm speeds up pricing of financial derivatives.

Algorithm efficiently learns deep ReLU networks with polynomial runtime in depth and parameters.

ReLU networks don't exponentially distort curve lengths as previously thought.

We show that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all kn…

New findings on depth vs. width in neural networks, showing depth can improve learnability.

We study the dynamics of gradient descent on objective functions of the form $f(\prod_{i=1}^{k} w_i)$ (with respect to scalar parameters $w_1,\ldots,w_k$), which arise in the context of training depth-$k$ linear neural networks. We prove that for standard random initializations, and under mild assumptions on $f$, the n…

In this paper we study the learnability of deep random networks from both theoretical and practical points of view. On the theoretical front, we show that the learnability of random deep networks with sign activation drops exponentially with its depth. On the practical front, we find that the learnability drops sharply…

Study shows depth improves generalization in deep learning models.

Let $f:\mathbb{S}^{d-1}\times \mathbb{S}^{d-1}\to\mathbb{S}$ be a function of the form $f(\mathbf{x},\mathbf{x}') = g(\langle\mathbf{x},\mathbf{x}'\rangle)$ for $g:[-1,1]\to \mathbb{R}$. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate $f…

Develops privacy-preserving multivariate median estimation methods.

Deep architecture such as hierarchical semi-Markov models is an important class of models for nested sequential data. Current exact inference schemes either cost cubic time in sequence length, or exponential time in model depth. These costs are prohibitive for large-scale problems with arbitrary length and depth. In th…

Monotone neural networks can approximate and interpolate functions efficiently.

Generates infinite-depth hierarchical clusters from few examples.

Study liquidity provision with exogenous competition using a reference market maker.

Existing depth separation results for constant-depth networks essentially show that certain radial functions in $\mathbb{R}^d$, which can be easily approximated with depth $3$ networks, cannot be approximated by depth $2$ networks, even up to constant accuracy, unless their size is exponential in $d$. However, the func…

Existing Rademacher complexity bounds for neural networks rely only on norm control of the weight matrices and depend exponentially on depth via a product of the matrix norms. Lower bounds show that this exponential dependence on depth is unavoidable when no additional properties of the training data are considered. We…

Rational neural networks approximate functions more efficiently with less depth.

This work generalizes bounds on the number of linear regions in CPWL NNs.

New $t$-LC triangulated manifolds are exponentially many.

Improved neural network depth-width trade-offs via dynamical systems.

Study shows depth improves trainability of neural networks by improving kernel conditioning.

Decision tree learning heuristics fail even in smoothed analysis for complex targets.

Study fractal and regular geometry in deep neural networks.

ForestPrune optimizes tree ensemble pruning for compactness and speed.

A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditi…

Study reveals how depth of reasoning affects generalization in models.

ResNets approximate log-Gaussian at initialization, improving network performance.

For any positive integer $k$, there exist neural networks with $Θ(k^3)$ layers, $Θ(1)$ nodes per layer, and $Θ(1)$ distinct parameters which can not be approximated by networks with $\mathcal{O}(k)$ layers unless they are exponentially large --- they must possess $Ω(2^k)$ nodes. This result is proved here for a class o…

Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.

For linear classifiers, the relationship between (normalized) output margin and generalization is captured in a clear and simple bound -- a large output margin implies good generalization. Unfortunately, for deep models, this relationship is less clear: existing analyses of the output margin give complicated bounds whi…

Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory. In a seminal paper, Telgarsky highlig…

Quantum algorithms simulate and exponentiate correlated Gaussian vectors for financial modeling.

Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency --- a shallow network of exponentially lar…

Signature volatility models are analyzed for existence, arbitrage, completeness, and hedging-error decomposition.

Analytical solution found for a three-layer network with a specific activation function.

A new complexity measure for neural networks improves upon classical methods.

Hyperbolic space outperforms Euclidean in learning hierarchical data.

AdaEnsemble learns adaptive feature interactions for CTR prediction.

Graphs avoid oversmoothing with properly initialized weights.

Quantum networks offer exponential communication savings for large machine learning models.

We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models c…

New bounds adaptively control spectral complexity of trained Transformers.