In this paper, we prove that a shallow neural network with a monotone sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function can arbitrarily well approximate any L^p(p>=2) integrable functions defined on R*[0,1]^n. We also prove that a shallow neural network with a sigmoid, ReLU, ELU, Softplus, or LeakyReLU act…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
10 results for “LeakyReLU”
Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width Onecs.LG
Residual networks with block width max(d_x, d_y) approximate all functions.
problem Achieving universal approximation with residual networks.
method Established bounds on block width for different activation functions.
result Minimum block width for universal approximation is max(d_x, d_y) with inner width 1.
We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz constant of the input-output relation. To that end, we first establish a global bound for the Lipschitz constant of …
New activation function BrownianReLU improves LSTM network performance on financial time series.
problem Gradient instability in noisy financial time series data.
method Introduces BrownianReLU, a stochastic activation function based on Brownian motion.
result Significantly improved predictive accuracy and generalization on financial datasets.
Flowification enriches neural networks with an inverse pass and likelihood monitoring.
problem Neural networks lack an inverse pass and likelihood monitoring, limiting their generative capabilities.
method Introduce flowification, enriching neural networks with a stochastic inverse pass and likelihood monitoring.
result Certain neural network architectures can be enriched to fall under the generalized notion of a normalizing flow.
GraN-GAN normalizes gradients for better GAN performance.
problem Improving image generation in GANs with piecewise linear discriminators.
method Piecewise Gradient Normalization (GraN) for input-dependent normalization.
result Significant performance gains in image generation across various datasets.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.
problem The issue of oversmoothing in attention-based GNNs.
method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.
result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.
In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal s…
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.