arXiv research

New metric space for ReLU codes connects to network safety and robustness.

Neural networks encode inputs deterministically and categorically, behaving like hash encoders.

Quantile regression with ReLU networks achieves minimax rates for various function types.

We apply neural nets with ReLU gates in online reinforcement learning. Our goal is to train these networks in an incremental manner, without the computationally expensive experience replay. By studying how individual neural nodes behave in online training, we recognize that the global nature of ReLU gates can cause und…

Model-based neural networks generalize better than ReLU networks for sparse recovery.

Activation functions have a notorious impact on neural networks on both training and testing the models against the desired problem. Currently, the most used activation function is the Rectified Linear Unit (ReLU). This paper introduces a new and novel activation function, closely related with the new activation $Swish…

In "Dictionary Learning" one tries to recover incoherent matrices $A^* \in \mathbb{R}^{n \times h}$ (typically overcomplete and whose columns are assumed to be normalized) and sparse vectors $x^* \in \mathbb{R}^h$ with a small support of size $h^p$ for some $0 <p < 1$ while having access to observations $y \in \mathbb{…

An artificial neuron is modelled as a weighted summation followed by an activation function which determines its output. A wide variety of activation functions such as rectified linear units (ReLU), leaky-ReLU, Swish, MISH, etc. have been explored in the literature. In this short paper, we explore what happens when the…

We present new families of continuous piecewise linear (CPWL) functions in Rn having a number of affine pieces growing exponentially in $n$. We show that these functions can be seen as the high-dimensional generalization of the triangle wave function used by Telgarsky in 2016. We prove that they can be computed by ReLU…

Guarantees sparse recovery for neural networks with iterative hard thresholding.

Improved code translation by preserving structure with composed fine-tuning.

Paper proposes faster certified robust training methods with short warmup.

SOC-ICNN expands neural network representational capacity by using conic optimization.

This study connects ReLU neural networks to toric geometry to analyze function realization.

We consider a deep ReLU / Leaky ReLU student network trained from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). The student network is \emph{over-realized}: at each layer $l$, the number $n_l$ of student nodes is more than that ($m_l$) of teacher. Under mild conditions…

We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks…

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

Large deviation principle for deep neural networks with ReLU activation.

Study shows efficient neural network approach for stochastic bandits.

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in…

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

Proves existence of optimal shallow neural networks with ReLU activation.

Gradient descent learns ReLU functions with non-zero bias efficiently.

This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem. We show that a subgradient descent algorithm, with random initialization, can provably recover orthogonal dictionaries on a natural nonsmooth, nonconvex $\ell_1$ minimization formulation of the problem, under mi…

Deep ReLU networks can be simplified to a three-layer model.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Kernel sparsity ("dying ReLUs") and lack of diversity are commonly observed in CNN kernels, which decreases model capacity. Drawing inspiration from information theory and wireless communications, we demonstrate the intersection of coding theory and deep learning through the Grassmannian subspace packing problem in CNN…

Modified ReLU networks improve regression estimation rates.

Bayesian ReLU nets fix asymptotic overconfidence with infinite features.

Improved robustness for deep neural networks with tighter bounds and attacks.

We consider the problem of computing the best-fitting ReLU with respect to square-loss on a training set when the examples have been drawn according to a spherical Gaussian distribution (the labels can be arbitrary). Let $\mathsf{opt} < 1$ be the population loss of the best-fitting ReLU. We prove: 1. Finding a ReLU wit…

Improved bounds on neural network expressivity.

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.

Framework explains how dual deep networks learn features from unlabeled data.

Transformers use ReLUs to approximate softmax efficiently.

PHP connects to ReLU neural networks for scalable Bayesian inference.

We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…

Paper analyzes GLM-tron for high-dimensional ReLU regression, providing upper and lower bounds.

The paper explores how ReLU DNNs can represent MPC policies and vice versa.

We study algebraic varieties of ReLU networks to understand their representable functions.

Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.

The paper calculates upper bounds on ReLU network Lipschitz constants.

Study approximates nonlinear functionals using deep ReLU networks.

New maximum score estimators using ReLU functions and deep neural networks.

Bayesian approach fixes overconfidence in ReLU networks, even slightly.

Neural networks with REctified Linear Unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being e.g. Gaussian, or require the network size an…