arXiv research

New findings on depth vs. width in neural networks, showing depth can improve learnability.

Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.

Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.

Existing depth separation results for constant-depth networks essentially show that certain radial functions in $\mathbb{R}^d$, which can be easily approximated with depth $3$ networks, cannot be approximated by depth $2$ networks, even up to constant accuracy, unless their size is exponential in $d$. However, the func…

Study shows depth improves generalization in deep learning models.

Transformers can learn noisy linear systems with depth and IID data.

The paper proves barriers to approximating functions with small weights and depth in neural networks.

Generative model separates objects, positions, and appearances in images.

This paper extends depth separation results to piece-wise oscillatory functions.

Let $f:\mathbb{S}^{d-1}\times \mathbb{S}^{d-1}\to\mathbb{S}$ be a function of the form $f(\mathbf{x},\mathbf{x}') = g(\langle\mathbf{x},\mathbf{x}'\rangle)$ for $g:[-1,1]\to \mathbb{R}$. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate $f…

New algorithm quantifies uncertainty in regression models for complex data types.

Poly-GNNs achieve similar performance regardless of depth, highlighting graph noise's dominance.

This paper explains a mechanism called phase collapse that improves image classification accuracy.

Deep RNNs excel at capturing long-term dependencies in sequential data.

New model separates objects in scenes, enabling novel arrangements and depth.

Following the seminal idea of Tukey, data depth is a function that measures how close an arbitrary point of the space is located to an implicitly defined center of a data cloud. Having undergone theoretical and computational developments, it is now employed in numerous applications with classification being the most po…

The expressivity of neural networks as a function of their depth, width and type of activation units has been an important question in deep learning theory. Recently, depth separation results for ReLU networks were obtained via a new connection with dynamical systems, using a generalized notion of fixed points of a con…

This work establishes universality for deep equivariant networks, overcoming limitations of previous approaches.

We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-lin…

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

We develop a scalable method for Bayesian neural networks with stochastic differential equations.

New findings show depth is more important than width in neural networks.

This work briefly explores the possibility of approximating spatial distance (alternatively, similarity) between data points using the Isolation Forest method envisioned for outlier detection. The logic is similar to that of isolation: the more similar or closer two points are, the more random splits it will take to se…

This paper explores how neural network width and depth behave as they approach infinity.

A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditi…

Deep networks can perfectly classify two low-dimensional manifolds on a sphere with large depth and width.

Deep networks can learn functions approximated by shallow networks, but not all functions.

Study reveals how depth of reasoning affects generalization in models.

This paper optimizes ReLU networks for approximating Hölder continuous functions.

For any positive integer $k$, there exist neural networks with $Θ(k^3)$ layers, $Θ(1)$ nodes per layer, and $Θ(1)$ distinct parameters which can not be approximated by networks with $\mathcal{O}(k)$ layers unless they are exponentially large --- they must possess $Ω(2^k)$ nodes. This result is proved here for a class o…

Neural network models are resource hungry. It is difficult to deploy such deep networks on devices with limited resources, like smart wearables, cellphones, drones, and autonomous vehicles. Low bit quantization such as binary and ternary quantization is a common approach to alleviate this resource requirements. Ternary…

Many results in recent years established polynomial time learnability of various models via neural networks algorithms. However, unless the model is linear separable, or the activation is a polynomial, these results require very large networks -- much more than what is needed for the mere existence of a good predictor.…

Most of the research in convolutional neural networks has focused on increasing network depth to improve accuracy, resulting in a massive number of parameters which restricts the trained network to platforms with memory and processing constraints. We propose to modify the structure of the Very Deep Convolutional Neural…

We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…

Measures difficulty of predictions to improve deep learning models.

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

Theory explains deep nonlinear networks' plateaus and transitions.

Generates infinite-depth hierarchical clusters from few examples.

Understanding the power of depth in feed-forward neural networks is an ongoing challenge in the field of deep learning theory. While current works account for the importance of depth for the expressive power of neural-networks, it remains an open question whether these benefits are exploited during a gradient-based opt…

Study on feature learning in Leaky ResNets, explaining bottleneck structure.

A new statistical concept, lepto-variance, is defined for stock returns using Regression Trees.

Non-negative matrix factorization is a problem of dimensionality reduction and source separation of data that has been widely used in many fields since it was studied in depth in 1999 by Lee and Seung, including in compression of data, document clustering, processing of audio spectrograms and astronomy. In this work we…

New bounds adaptively control spectral complexity of trained Transformers.

Independent component analysis (ICA) has been widely used for blind source separation in many fields such as brain imaging analysis, signal processing and telecommunication. Many statistical techniques based on M-estimates have been proposed for estimating the mixing matrix. Recently, several nonparametric methods have…

This paper studies Fenchel-Young losses, a generic way to construct convex loss functions from a regularization function. We analyze their properties in depth, showing that they unify many well-known loss functions and allow to create useful new ones easily. Fenchel-Young losses constructed from a generalized entropy, …

New bounds improve graph node classification using optimal transport.

While on some natural distributions, neural-networks are trained efficiently using gradient-based algorithms, it is known that learning them is computationally hard in the worst-case. To separate hard from easy to learn distributions, we observe the property of local correlation: correlation between local patterns of t…

Enhanced ODT with Feature Concatenation boosts learning efficiency.