arXiv research

Model-based neural networks generalize better than ReLU networks for sparse recovery.

ReLU networks learn simple models even with many parameters, overcoming traditional wisdom.

This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected traject…

Guarantees sparse recovery for neural networks with iterative hard thresholding.

Paper reveals hidden convexities in deep learning models using sparse signal processing.

This paper finds sparsest ReLU networks for interpolating data.

Improved bounds for neural network approximations of functions.

A new method uses Coordinate Descent to optimize ResNet networks for private inference.

Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, w…

This work shows how penalising bias terms in norm regularisation leads to sparse solutions.

Sparse connectivity improves generalization in neural networks below the Edge of Stability.

Deep ReLU networks can efficiently approximate Sobolev and Besov functions.

Neural networks can approximate positive homogeneous functions, especially with multiple hidden layers.

Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to $\log n$-factors) under a general composition assumption on the re…

Paper analyzes how neural networks learn from a teacher in a specific setting.

Whereas recovery of the manifold from data is a well-studied topic, approximation rates for functions defined on manifolds are less known. In this work, we study a regression problem with inputs on a $d^*$-dimensional manifold that is embedded into a space with potentially much larger ambient dimension. It is shown tha…

Curriculum learning helps neural networks learn parities more efficiently.

Deep ReLU networks approximate functions in Sobolev and Besov spaces efficiently.

The paper tightens bounds on covering numbers for deep ReLU networks.

Deep neural networks with specific parameter sets can approximate smooth functions efficiently.

Deep neural networks improve mean function estimation for functional data.

This work studies how contrastive learning extracts features from unlabeled data.

The angular power spectrum characterizes neural network complexity.

There has been a large amount of interest, both in the past and particularly recently, into the power of different families of universal approximators, e.g. ReLU networks, polynomials, rational functions. However, current research has focused almost exclusively on understanding this problem in a worst-case setting, e.g…

Rectified linear units, or ReLUs, have become the preferred activation function for artificial neural networks. In this paper we consider two basic learning problems assuming that the underlying data follow a generative model based on a ReLU-network -- a neural network with ReLU activations. As a primarily theoretical …

This study connects ReLU neural networks to toric geometry to analyze function realization.

New method reduces over-parametrization in neural networks, ensuring sparsity and finite network size.

We discuss approximation of functions using deep neural nets. Given a function $f$ on a $d$-dimensional manifold $Γ\subset \mathbb{R}^m$, we construct a sparsely-connected depth-4 neural network and bound its error in approximating $f$. The size of the network depends on dimension and curvature of the manifold $Γ$, the…

QSD enhances deep network performance through biologically plausible dropout.

Spike-and-Slab Deep Learning (SS-DL) is a fully Bayesian alternative to Dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with unknown levels of smoothness. Indeed, we show that the posterior distribution concentrates at t…

Optimal sketching bounds for sparse linear regression under various loss functions are established.

Deep ReLU networks can be simplified to a three-layer model.

Theoretical analysis of deep neural networks for time series data.

New theory shows large learning rates prevent overfitting in neural networks.

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

Large deviation principle for deep neural networks with ReLU activation.

Proves existence of optimal shallow neural networks with ReLU activation.

Modified ReLU networks improve regression estimation rates.

Deep neural networks (DNNs) have emerged as key enablers of machine learning. Applying larger DNNs to more diverse applications is an important challenge. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more layers a…

We consider the problem of computing the best-fitting ReLU with respect to square-loss on a training set when the examples have been drawn according to a spherical Gaussian distribution (the labels can be arbitrary). Let $\mathsf{opt} < 1$ be the population loss of the best-fitting ReLU. We prove: 1. Finding a ReLU wit…

Early alignment in neural networks leads to sparse representations but hinders convergence.

Paper analyzes approximation and learning of MoEs with (P)ReLU activation.

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

Improved bounds on neural network expressivity.

TILT improves target domain performance by penalizing an auxiliary component on unlabeled target inputs.

Study shows efficient neural network approach for stochastic bandits.

PHP connects to ReLU neural networks for scalable Bayesian inference.

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.