arXiv research

Improved sample complexity for ReLU networks with norm constraints.

New algorithms optimize neural networks with ReLU activations using sampling.

New study shows neural networks need many samples for training.

New findings on depth vs. width in neural networks, showing depth can improve learnability.

Paper analyzes GLM-tron for high-dimensional ReLU regression, providing upper and lower bounds.

Gradient descent learns ReLU functions with non-zero bias efficiently.

New method for symmetric matrix completion using ReLU sampling.

Real world data often exhibit low-dimensional geometric structures, and can be viewed as samples near a low-dimensional manifold. This paper studies nonparametric regression of Hölder functions on low-dimensional manifolds using deep ReLU networks. Suppose $n$ training data are sampled from a Hölder function in $\mathc…

Learning ReLU networks to high uniform accuracy requires exponentially many samples.

New theory for local parameterization of deep ReLU networks.

We establish $L^{\infty}$ and $L^2$ error bounds for functions of many variables that are approximated by linear combinations of ReLU (rectified linear unit) and squared ReLU ridge functions with $\ell^1$ and $\ell^0$ controls on their inner and outer parameters. With the squared ReLU ridge function, we show th…

Paper analyzes sample complexity for offline RL with deep ReLU networks.

We propose and analyze a new family of algorithms for training neural networks with ReLU activations. Our algorithms are based on the technique of alternating minimization: estimating the activation patterns of each ReLU for all given samples, interleaved with weight updates via a least-squares step. The main focus of …

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

Polynomial time algorithm learns depth-2 neural networks with ReLU activations.

Gradient flow on ReLU networks converges to a simple model with few regions.

Echo state networks with random weights can approximate any continuous system.

It has been noted in existing literature that over-parameterization in ReLU networks generally improves performance. While there could be several factors involved behind this, we prove some desirable theoretical properties at initialization which may be enjoyed by ReLU networks. Specifically, it is known that He initia…

Bayesian neural networks show good correlation between out-of-sample performance and Bayesian evidence.

Bayesian free energy remains bounded for deep ReLU networks in overparametrized cases.

Neural networks with REctified Linear Unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being e.g. Gaussian, or require the network size an…

Paper optimizes private ReLU regression with relaxed assumptions.

This paper examines how noise affects deep neural networks and improves their performance.

New study shows exponential sample growth for ReQU neural networks.

In this paper, we consider parameter recovery for non-overlapping convolutional neural networks (CNNs) with multiple kernels. We show that when the inputs follow Gaussian distribution and the sample size is sufficiently large, the squared loss of such CNNs is $\mathit{~locally~strongly~convex}$ in a basin of attraction…

Gradient descent methods for deep ReLU networks achieve optimal generalization rates.

New measure FTC quantifies how much a ReLU network can fine-tune.

Polynomial-time convex optimization for CNNs with ReLU activations.

Algorithm learns two-layer residual units using ReLU activations from samples.

Deep ReLU networks generalize well with few parameters.

ReLU networks learn simple models even with many parameters, overcoming traditional wisdom.

New property ensures neural networks generalize well with limited data.

Gradient descent learns over-param neural nets better than NTK.

We study the sample complexity of learning one-hidden-layer convolutional neural networks (CNNs) with non-overlapping filters. We propose a novel algorithm called approximate gradient descent for training CNNs, and show that, with high probability, the proposed algorithm with random initialization grants a linear conve…

Model-based neural networks generalize better than ReLU networks for sparse recovery.

Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to $\log n$-factors) under a general composition assumption on the re…

Study identifies three quantization regimes for ReLU networks.

Unified framework reveals regularization mechanism in deep ReLU networks via convex optimization.

We study the problem of learning one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian distribution and the outputs are generated from a noisy teacher network. We analyze the performance of gradient descent for training such kind of n…

Develops exact convex optimization formulations for neural networks.

Gradient flow of ReLU networks converges in low-correlation high-dimensional data.

We consider the problem of estimating the parameters of a $d$-dimensional rectified Gaussian distribution from i.i.d. samples. A rectified Gaussian distribution is defined by passing a standard Gaussian distribution through a one-layer ReLU neural network. We give a simple algorithm to estimate the parameters (i.e., th…

New polynomial-time solutions found for training ReLU networks, mirroring Max-Cut complexity.

Before training a neural net, a classic rule of thumb is to randomly initialize the weights so the variance of activations is preserved across layers. This is traditionally interpreted using the total variance due to randomness in both weights \emph{and} samples. Alternatively, one can interpret the rule of thumb as pr…

While neural networks have achieved high performance in different learning tasks, their accuracy drops significantly in the presence of small adversarial perturbations to inputs. Defenses based on regularization and adversarial training are often followed by new attacks to defeat them. In this paper, we propose attack-…

New algorithm learns ReLU networks efficiently using Schur polynomials.

Residual neural networks don't help overcome sampling complexity issues.

Study on benign overfitting in leaky ReLUs with moderate input dimensions.