arXiv research

Estimates nonlinear Hawkes processes using RKHSs with ReLU rectification.

The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tra…

A new method detects and corrects outliers using optimal transport.

RTSCV detects unknown unknowns to improve model performance.

Derives conformal parameters of curves using inscribed circular polygons.

AReLU uses attention-based rectification to improve neural network performance.

We present new rectification theorems of degenerate quasi-conformal structures that give a meaning to quotients of Riemann surfaces with empty interior "fundamental domains". These techniques are used to define the unique renormalization of polynomials with Cantor set Julia sets.

The Mutual Information (MI) is an often used measure of dependency between two random variables utilized in information theory, statistics and machine learning. Recently several MI estimators have been proposed that can achieve parametric MSE convergence rate. However, most of the previously proposed estimators have th…

We introduce a new neural network model, together with a tractable and monotone online learning algorithm. Our model describes feed-forward networks for classification, with one output node for each class. The only nonlinear operation is rectification using a ReLU function with a bias. However, there is a rectifier on …

Consider a bundle of circles passing through 0 in 4-dimensional space. It is said to be rectifiable if there is a germ of diffeomorphism at 0 that takes all circles from our bundle to straight lines. We will give a classification of all rectifiable bundles of circles containing sufficiently many circles in general posi…

In this work we compute lower Lipschitz bounds of $\ell_p$ pooling operators for $p=1, 2, \infty$ as well as $\ell_p$ pooling operators preceded by half-rectification layers. These give sufficient conditions for the design of invertible neural network layers. Numerical experiments on MNIST and image patches confirm tha…

Recently, the geodesibility of planar vector fields, which are algebrizable (differentiable in the sense of Lorch for some associative and commutative unital algebra), has been established. In this paper, we consider algebrizable three-dimensional vector fields, for which we give rectifications and Riemannian metrics u…

This study connects ReLU neural networks to toric geometry to analyze function realization.

Poisson variational autoencoders introduce a metabolic cost term that penalizes high baseline activity.

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

Large deviation principle for deep neural networks with ReLU activation.

We study the supremum of the volume of hyperbolic polyhedra with some fixed combinatorics and with vertices of any kind (real, ideal or hyperideal). We find that the supremum is always equal to the volume of the rectification of the 1-skeleton. The theorem is proved by applying a sort of volume-increasing flow to any h…

Study shows efficient neural network approach for stochastic bandits.

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in…

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

Proves existence of optimal shallow neural networks with ReLU activation.

Gradient descent learns ReLU functions with non-zero bias efficiently.

Deep ReLU networks can be simplified to a three-layer model.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Modified ReLU networks improve regression estimation rates.

Bayesian ReLU nets fix asymptotic overconfidence with infinite features.

The goal of this paper is to describe all local diffeomorphisms mapping a family of circles, in an open subset of $\r^3$, into straight lines. This paper contains two main results. The first is a complete description of the rectifiable collection of circles in $\r^3$ passing through one point. It turns out that to be r…

We consider the problem of computing the best-fitting ReLU with respect to square-loss on a training set when the examples have been drawn according to a spherical Gaussian distribution (the labels can be arbitrary). Let $\mathsf{opt} < 1$ be the population loss of the best-fitting ReLU. We prove: 1. Finding a ReLU wit…

Improved bounds on neural network expressivity.

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.

Transformers use ReLUs to approximate softmax efficiently.

PHP connects to ReLU neural networks for scalable Bayesian inference.

We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…

Paper analyzes GLM-tron for high-dimensional ReLU regression, providing upper and lower bounds.

The paper explores how ReLU DNNs can represent MPC policies and vice versa.

We study algebraic varieties of ReLU networks to understand their representable functions.

Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.

The paper calculates upper bounds on ReLU network Lipschitz constants.

Study approximates nonlinear functionals using deep ReLU networks.

New maximum score estimators using ReLU functions and deep neural networks.

Bayesian approach fixes overconfidence in ReLU networks, even slightly.

Neural networks with REctified Linear Unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being e.g. Gaussian, or require the network size an…

Gradient descent biases towards stable rank networks for nearly-orthogonal data.

The paper reveals how clustering in ReLU networks enhances interpretability.

Leaky ReLU activations improve the calibration of Bayesian neural networks.

Polynomial time algorithm learns depth-2 neural networks with ReLU activations.