Recently, self-normalizing neural networks (SNNs) have been proposed with the intention to avoid batch or weight normalization. The key step in SNNs is to properly scale the exponential linear unit (referred to as SELU) to inherently incorporate normalization based on central limit theory. SELU is a monotonically incre…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
10 results for “SELU”
Improved speech recognition with faster training and inference.
problem Training very deep CNNs for speech recognition is difficult.
method Proposed SNDCNN using SELU activations instead of RELU and shortcut connections/BN.
result Achieved similar or lower WER with faster training and inference.
Modelling of daily reference evapotranspiration using deep neural network in different climatescs.LG
Study uses DNN to accurately estimate daily ET o in various climates.
problem Precise estimation of reference evapotranspiration (ET o ) for irrigation and water management.
method Investigated artificial neural network (ANN) and deep neural network (DNN) models using six meteorological inputs.
result DNN models, especially P-DNN-SeLU, achieve high accuracy in daily ET o estimation.
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.
Goldilocks activation functions improve neural network performance.
problem Improving neural network performance and understanding signal transformation.
method Introducing Goldilocks activation functions that locally deform input signals.
result Goldilocks networks outperform or match SELU and RELU on CIFAR-10 and CIFAR-100 datasets.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networkscs.NE
Activation functions influence behavior and performance of DNNs. Nonlinear activation functions, like Rectified Linear Units (ReLU), Exponential Linear Units (ELU) and Scaled Exponential Linear Units (SELU), outperform the linear counterparts. However, selecting an appropriate activation function is a challenging probl…
Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot …
A novel metric and framework for evaluating gradient norm equality in deep neural networks.
problem Evaluation of gradient norm equality in complex DNNs requires strong assumptions or complex analysis.
method Proposes a novel metric called Block Dynamical Isometry and a modularized statistical framework based on free probability.
result Gradient Norm Equality is a universal philosophy behind initialization, normalization, and network structures.
It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume…