arXiv research

New polynomial-time solutions found for training ReLU networks, mirroring Max-Cut complexity.

Polynomial-time algorithm learns ReLU networks without assumptions.

Polynomial time algorithm learns depth-2 neural networks with ReLU activations.

We study algebraic varieties of ReLU networks to understand their representable functions.

Polynomial-time convex optimization for CNNs with ReLU activations.

New algorithm learns ReLU networks efficiently using Schur polynomials.

Algorithm learns arbitrary ReLU neurons under Gaussian inputs.

New algorithm for learning ReLU networks with Gaussian noise, improving previous results.

Researchers create integral representations for two-layer ReLU networks with quantitative bounds.

Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.

There has been a large amount of interest, both in the past and particularly recently, into the power of different families of universal approximators, e.g. ReLU networks, polynomials, rational functions. However, current research has focused almost exclusively on understanding this problem in a worst-case setting, e.g…

New findings on depth vs. width in neural networks, showing depth can improve learnability.

Gradient descent learns ReLU functions with non-zero bias efficiently.

We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial in the number of training samples and the number of hidden neurons. Our theory utilizes semi-infinite duality and minimum norm regularization…

We give the first dimension-efficient algorithms for learning Rectified Linear Units (ReLUs), which are functions of the form $\mathbf{x} \mapsto \max(0, \mathbf{w} \cdot \mathbf{x})$ with $\mathbf{w} \in \mathbb{S}^{n-1}$. Our algorithm works in the challenging Reliable Agnostic learning model of Kalai, Kanade, and Ma…

Gradient descent learns over-param neural nets better than NTK.

Hardness proven for learning neural networks with polynomial size and Gaussian inputs.

Path regularization reveals convex optimization in deep ReLU networks.

Algorithm efficiently learns deep ReLU networks with polynomial runtime in depth and parameters.

The paper analyzes a simple neural network model with algebraic methods.

In this paper, we consider parameter recovery for non-overlapping convolutional neural networks (CNNs) with multiple kernels. We show that when the inputs follow Gaussian distribution and the sample size is sufficiently large, the squared loss of such CNNs is $\mathit{~locally~strongly~convex}$ in a basin of attraction…

Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree $O(\text{polylog}(1/ε))$ which is $ε$-close, and similarly for any rational function there exists a ReLU network of size $O(\text{polylog}(1/ε…

We consider the problem of computing the best-fitting ReLU with respect to square-loss on a training set when the examples have been drawn according to a spherical Gaussian distribution (the labels can be arbitrary). Let $\mathsf{opt} < 1$ be the population loss of the best-fitting ReLU. We prove: 1. Finding a ReLU wit…

New bounds show complex neural networks need many queries to learn.

The paper proves barriers to approximating functions with small weights and depth in neural networks.

Polynomial-time reachability for LTI systems with TLL NN controllers is achieved.

Gradient descent methods for deep ReLU networks achieve optimal generalization rates.

Neural networks with ReLU^k approximate Sobolev functions efficiently via Radon transform.

We solve ReLU regression with efficient approximations for various distributions.

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N…

Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.

For any positive integer $k$, there exist neural networks with $Θ(k^3)$ layers, $Θ(1)$ nodes per layer, and $Θ(1)$ distinct parameters which can not be approximated by networks with $\mathcal{O}(k)$ layers unless they are exponentially large --- they must possess $Ω(2^k)$ nodes. This result is proved here for a class o…

Rectified Linear Units (ReLUs) are among the most widely used activation function in a broad variety of tasks in vision. Recent theoretical results suggest that despite their excellent practical performance, in various cases, a substitution with basis expansions (e.g., polynomials) can yield significant benefits from b…

Training neural networks is hard in fixed dimensions.

In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to *global optimality* with runtime polynomial in the data size albeit exponential in the input dimension. Further, we impro…

This paper extends the convergence rate of DEQs with ReLU to any general activation.

Characterizes neural kernel and NNGP for various activations.

Deep neural networks struggle with numerical instability during training.

Existence of minimizers proven for residual ANNs with ReLU activation.

New insights into training ReLU networks, especially as data dimensionality increases.

Unified framework reveals regularization mechanism in deep ReLU networks via convex optimization.

Neural networks solve copositive programs, revealing insights into training problems.

A wide variety of activation functions have been proposed for neural networks. The Rectified Linear Unit (ReLU) is especially popular today. There are many practical reasons that motivate the use of the ReLU. This paper provides new theoretical characterizations that support the use of the ReLU, its variants such as th…

Wide networks with polynomial activations have proven asymptotic behavior.

In this work, we examine the process of Tropical Polynomial Division, a geometric method which seeks to emulate the division of regular polynomials, when applied to those of the max-plus semiring. This is done via the approximation of the Newton Polytope of the dividend polynomial by that of the divisor. This process i…

New findings suggest minimax optimality doesn't guarantee distribution learning for GANs.

Neural networks can approximate high-dimensional classifiers with ReLU networks under margin conditions.

We present a new, unifying approach following some recent developments on the complexity of neural networks with piecewise linear activations. We treat neural network layers with piecewise linear activations as tropical polynomials, which generalize polynomials in the so-called $(\max, +)$ or tropical algebra, with pos…