arXiv research

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in…

Recently, neural networks in machine learning use rectified linear units (ReLUs) in early processing layers for better performance. Training these structures sometimes results in "dying ReLU units" with near-zero outputs. We first explore this condition via simulation using the CIFAR-10 dataset and variants of two popu…

New activation function BrownianReLU improves LSTM network performance on financial time series.

Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, w…

Kernel sparsity ("dying ReLUs") and lack of diversity are commonly observed in CNN kernels, which decreases model capacity. Drawing inspiration from information theory and wireless communications, we demonstrate the intersection of coding theory and deep learning through the Grassmannian subspace packing problem in CNN…

DyS model improves survival analysis accuracy and interpretability.

In this note, we study the integral of the 1-form $\log x\frac{dy}{y}-\log y\frac{dx}{x}$ over certain plane curves defined by A-polynomials of knots. It is quite surprising that a Chern-Simons type invariant of 3-manifolds, which can be geometrically computed, may be used to get the exact values of those integrals. Th…

We study a variant of decision-theoretic online learning in which the set of experts that are available to Learner can shrink over time. This is a restricted version of the well-studied sleeping experts problem, itself a generalization of the fundamental game of prediction with expert advice. Similar to many works in t…

Study of nonlocal mean curvature and symmetry of surfaces.

We prove that, starting at an initial metric $g(0)=e^{2u_0}(dx^2+dy^2)$ on $\mathbb{R}^2$ with bounded scalar curvature and bounded $u_0$, the Ricci flow $\partial_t g(t)=-R_{g(t)}g(t)$ converges to a flat metric on $\mathbb{R}^2$.

We explicitely compute the essential spectrum of the Laplace-Beltrami operator for $p$-forms for the class of warped product metrics $dσ^2= y^{2a}dy^2 + y^{2b}dθ_{\partial M}^2$, where $y$ is a boundary defining function on a compact manifold with boundary $M$.

The isotropic 3-space \mathbb{I}^{3} is a real affine 3-space endowed with the metric dx^{2}+dy^{2}. In this paper we describe Weingarten and linear Weingarten affine translation surfaces in \mathbb{I}^{3}. Further we classify the affine translation surfaces in \mathbb{I}^{3} that satisfy certain equations in terms of …

A semi-isotropic space is a real affine 3-space endowed with the non-degenerate metric dx^{2}-dy^{2}. The main purpose of this paper is to describe the surfaces of revolution in the semi-isotropic space that satisfy some equations in terms of the position vector and the Laplace operators with respect to the first and t…

This study connects ReLU neural networks to toric geometry to analyze function realization.

We introduce a new Self-Organized Criticality (SOC) model for simulating price evolution in an artificial financial market, based on a multilayer network of traders. The model also implements, in a quite realistic way with respect to previous studies, the order book dy- namics, by considering two assets with variable f…

Optimal transport theory characterizes convex order between probability measures.

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

New method tests independence using ROC analysis and bipartite ranking.

In this paper we characterize the degenerate elliptic equations F(D^2u)=0 whose viscosity subsolutions, (F(D^2u) \geq 0), satisfy the strong maximum principle. We introduce an easily computed function f(t) for t > 0, determined by F, and we show that the strong maximum principle holds depending on whether the integral …

Large deviation principle for deep neural networks with ReLU activation.

Study shows efficient neural network approach for stochastic bandits.

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

Proves existence of optimal shallow neural networks with ReLU activation.

We present an approach to cohomological dimension theory based on infinite symmetric products and on the general theory of dimension called the extension dimension. The notion of the extension dimension $\ExD(X)$ was introduced by A.N.Dranishnikov \cite {D$_5$} in the context of compact spaces and CW complexes. This pa…

Gradient descent learns ReLU functions with non-zero bias efficiently.

Deep ReLU networks can be simplified to a three-layer model.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Modified ReLU networks improve regression estimation rates.

Bayesian ReLU nets fix asymptotic overconfidence with infinite features.

Let $g=e^{2u}(dx^2+dy^2)$ be a conformal metric defined on the unit disk of $\mathbf{C}$. We give an estimate of $\|\nabla u\|_{L^{2,\infty}(D_\frac{1}{2})}$ when $\|K(g)\|_{L^1}$ is small and $\frac{μ(B_r^g(z),g)}{πr^2}<Λ$ for any $r$ and $z\in D_\frac{3}{4}$. Then we will use this estimate to study the Gromov-Hausdor…

We consider the problem of computing the best-fitting ReLU with respect to square-loss on a training set when the examples have been drawn according to a spherical Gaussian distribution (the labels can be arbitrary). Let $\mathsf{opt} < 1$ be the population loss of the best-fitting ReLU. We prove: 1. Finding a ReLU wit…

Improved bounds on neural network expressivity.

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.

Transformers use ReLUs to approximate softmax efficiently.

PHP connects to ReLU neural networks for scalable Bayesian inference.

We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…

The paper proves geodesics and conic sections are length-minimizing under specific metrics.

Paper analyzes GLM-tron for high-dimensional ReLU regression, providing upper and lower bounds.

The paper explores how ReLU DNNs can represent MPC policies and vice versa.

Study measures risk spillovers between US and China's agricultural futures markets.

We study algebraic varieties of ReLU networks to understand their representable functions.

Using the large deviation principle (LDP) for a re-scaled fractional Brownian motion $B^H_t$ where the rate function is defined via the reproducing kernel Hilbert space, we compute small-time asymptotics for a correlated fractional stochastic volatility model of the form $dS_t=S_tσ(Y_t) (\barρ dW_t +ρdB_t), \,dY_t=dB^H…

Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.

The paper calculates upper bounds on ReLU network Lipschitz constants.

Study approximates nonlinear functionals using deep ReLU networks.

New maximum score estimators using ReLU functions and deep neural networks.