The study evaluates different rectified activation functions in CNNs and finds RReLU superior.
problem Improving performance of CNNs using rectified activation functions.
method Compared standard ReLU, Leaky ReLU, PReLU, and randomized RReLU on image classification tasks.
result RReLU outperforms other rectified activation functions, achieving 75.68% accuracy on CIFAR-100.
Paper proposes a new activation function to reduce overfitting and large weight update issues.
problem Overfitting and large weight update problems in neural networks.
method Introduces a new activation function called Thresholded Exponential Rectified Linear Units (TERELU).
result TERELU shows better performance in reducing overfitting and large weight update issues compared to other activation functions.
Improved activation function NLReLU boosts neural network performance.
problem Performance issues with ReLU activation function.
method NLReLU uses parametric natural logarithmic transform to improve ReLU.
result NLReLU provides higher accuracy than ReLU in various neural networks.
AReLU uses attention-based rectification to improve neural network performance.
problem Improving neural network performance through better activation functions.
method Integrates attention mechanism with rectified linear unit (ReLU) to learn and scale feature maps.
result AReLU significantly boosts performance of most network architectures with minimal changes.
CRITS improves time series classification with interpretable local explanations.
problem Lack of detailed explanations in time series classification models.
method CRITS uses convolutional kernels, max-pooling, and rectified linear units to extract feature weights.
result CRITS provides intrinsically interpretable local explanations without requiring gradients or random perturbations.
Model stores many more patterns than neurons, improving pattern recognition.
problem Storing and retrieving many more patterns than neurons in a network.
method Constructs a family of models interpolating between feature-matching and prototype modes, corresponding to neural networks with various activation functions.
result Higher rectified polynomials can be used in neural networks for improved pattern recognition.
ABUs learn and adapt activation functions for deep neural networks.
problem Lack of a unified theory connecting task and network properties with activation functions.
method Introduce Adaptive Blending Units (ABUs) as a trainable linear combination of activation functions.
result Advantages of ABUs over common activation functions across various network specifications.
New insights into neural network kernels for rectified MLPs.
problem Understanding the behavior of weight distributions in rectified MLPs.
method Deriving equivalent kernels for MLPs with ReLU or Leaky ReLU activations under various weight distributions.
result Kernels corresponding to layers with certain weight distributions are asymptotically universal and well approximated by Gaussian kernels.
Dropout training improves neural networks' performance.
problem Improving neural network convergence and generalization.
method Two-layer neural networks with ReLU activations, overparametrization, and positive margin assumption.
result Dropout training achieves ε-suboptimality in test error in O(1/ε) iterations.
Proposes adjustable rectifiers for binary neural networks.
problem Training deep binary neural networks is challenging.
method Softly encourages activations to be binary approximating step functions.
result Can binarize all layers with minimal performance loss.
New activations improve deep network reproducibility without sacrificing accuracy.
problem Deep networks' reproducibility issues, especially on distributed systems.
method Developed SmeLU activations, smoother than ReLU, to enhance reproducibility.
result SmeLU activations provide better accuracy-reproducibility tradeoffs.
Empirical bounds estimate the number of linear regions in deep ReLU networks.
problem Estimating the number of linear regions in deep neural networks.
method Empirical bounds based on activation patterns and probabilistic inference.
result Fast proxy for the number of linear regions of deep neural networks.
Paper provides convergence rates for rectifier convnets.
problem Understanding why rectifier networks perform well empirically.
method Introduces gated games to capture rectifier units' gating function.
result Gradient descent on rectifier convnets converges to a critical point.
GOLS finds activation functions affect training robustness, especially ReLU.
problem Investigate how different activation functions impact GOLS in neural network training.
method Identify SNN-GPPs for GOLS, analyze activation function effects on gradient continuity.
result GOLS robust for most activation functions but sensitive to ReLU.
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural ne…
Paper provides convergence guarantees for rectifier networks using neural Taylor approximations.
problem Smoothness and convexity issues in modern convolutional networks.
method Neural Taylor approximations and Taylor loss for optimization.
result Guarantees match lower bounds for convex nonsmooth functions and accurately capture optimization dynamics.
Upper bound on knots in neural networks simplifies complexity analysis.
problem Understanding the complexity of neural networks with rectified linear units.
method Inductive analysis on the number of layers and neurons, proving an upper bound.
result A tight upper bound on the number of knots in neural networks is derived.
Reduced parameter space improves neural network training efficiency.
problem Training efficiency and optimization in neural networks.
method Training weights on a unit sphere and thresholds in a bounded interval.
result Equivalent performance with reduced parameter space.
New method learns neural network activation functions from data.
problem Learning activation functions for neural networks.
method Model each neuron's activation function as a small neural network.
result Learned activation functions improve network performance.
Study approximates nonlinear functionals using deep ReLU networks.
problem Approximating nonlinear continuous functionals with neural networks.
method Constructs continuous piecewise linear interpolation under simple triangulation, analyzes rates of approximation.
result Established rates of approximation for functional deep ReLU networks.
Convex relaxations improve CNNs with fixed weights.
problem Improving CNNs with fixed weights.
method Convex relaxations for CNNs with fixed weights using second order cone programs.
result The relaxation recovers the global minimum under a planted model assumption.
New activation functions mimic neuronal biology to improve deep learning performance.
problem Vanishing gradients and suboptimal learning in deep learning models.
method Introducing bionodal root unit (BRU) activation functions based on neuronal cell properties.
result BRU activation functions lead to faster training and better generalization in deep learning models.
New algorithm reveals piecewise affine structure of neural networks.
problem Lack of strong guarantees on deep neural networks' behavior in safety-critical applications.
method Developed a novel algorithm to compute the piecewise affine form of neural networks.
result Computed piecewise affine representations of neural networks with rectified linear unit activations.
DEUs learn nonlinear activation functions from data, reducing network size.
problem Fixed activation functions in neural networks limit performance.
method Differential equation units (DEUs) learn nonlinear activation functions from data.
result DEUs enable neurons to change their activation functions during training.
New neural network uses differential equations for adaptable activation functions.
problem Fixed activation functions limit neural network performance and size.
method Introduces differential equation units (DEUs) that learn nonlinear activation functions.
result DEUs enable more compact networks with comparable performance.
This paper corrects ReLU attribution and compares activation functions in deep learning.
problem Historical misattribution and performance comparison of activation functions.
method Historical tracing and empirical comparison of ReLU, Tanh, and Sigmoid across tasks.
result ReLU outperforms Sigmoid and Tanh in deep learning tasks.
A new activation function, E-swish, improves model accuracy on image classification tasks.
problem Improving activation functions for neural network performance.
method Introducing E-swish = βx * sigmoid(x) and comparing it to ReLU and Swish.
result E-swish outperforms ReLU and Swish in accuracy improvements on Cifar10 and Cifar100.
L*ReLU improves deep learning for fine-grained image classification.
problem Fine-grained image classification requires specific AFs.
method Proposes L*ReLU, piece-wise linear AFs for deep learning.
result L*ReLU achieves superior results on FGVC datasets.
Evolutionary algorithms improve neural network performance by discovering better activation functions.
problem The choice of activation function affects neural network performance, but ReLU remains dominant.
method Defined a tree-based search space of candidate activation functions and used evolutionary algorithms (mutation, crossover, exhaustive search) to explore and discover better functions.
result Replacing ReLU with evolved activation functions statistically significantly increases network accuracy.
Deep neural networks enforce non-crossing quantile regression curves.
problem Estimating quantile regression curves without crossing.
method Penalized deep ReQU neural networks with a non-crossing penalty.
result Established non-asymptotic risk and error bounds for the estimated QRP.
Study embeddings between Barron spaces with various activation functions, focusing on RePU.
problem Understanding the influence of activation functions on infinitely wide neural networks.
method Prove embeddings by constructing push-forward maps on measures representing functions.
result Barron spaces with RePU activation have a hierarchical structure similar to Sobolev spaces.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
Automatically discovers effective activation functions for deep learning.
problem Inconsistent performance of novel activation functions in deep learning networks.
method Evolutionary search for general form, gradient descent for parameters.
result Significant performance improvements over ReLU and other functions.
Adam optimizes DNNs to induce weight sparsity.
problem Large DNN models for edge devices.
method Using Adam optimizer with ReLU activations and L2 regularization.
result Deep learning automatically induces group sparsity of weights.
Enhanced GCNs with non-parametric activation functions outperform baselines.
problem Simple activation functions in GNNs limit model flexibility.
method Extended kernel activation function for GCNs, regularizable and smooth.
result Significant improvement over baseline GCNs with similar depth/size.
This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.
problem Exact data interpolation using sparse, infinitely wide neural networks.
method Atomic norm framework to derive convex hulls and equivalent convex formulations.
result Simple characterizations of convex hulls for different constraints on network weights and biases.
ReLU activations lead to smoother learning curves compared to sigmoidal activations in neural networks.
problem Comparing the performance of ReLU and sigmoidal activations in neural networks.
method Analytical computation of learning curves in shallow networks with different activation functions.
result ReLU networks exhibit continuous transitions in performance, while sigmoidal networks show discontinuous transitions.
Multivariate splines linked to infinitely-wide neural networks with improved numerical performance.
problem Understanding the relationship between multivariate splines and neural networks.
method Showed multivariate splines can be represented as random features in infinitely-wide neural networks with a homogeneous activation function.
result The function space of multivariate splines is a Sobolev space on a Euclidean ball with explicit norm bounds on derivatives.
Random neural networks with ReLU activations are non-Gaussian processes.
problem Understanding the behavior of neural networks with random initialization and rectified linear units.
method Proving these networks are non-Gaussian processes and deriving their properties.
result These networks can converge to non-Gaussian processes under certain conditions.
New framework explains deep neural networks using variational spline theory.
problem Understanding functions learned by deep neural networks.
method Developed a variational framework and function space.
result Deep ReLU networks are solutions to regularized data fitting problems over the proposed function space.
Rectifying submanifolds are characterized by their tangential position vector field component.
problem Characterizing rectifying submanifolds in Euclidean spaces.
method Introducing rectifying submanifolds and proving their properties.
result Rectifying submanifolds are identified by a specific tangential vector field property.
New method trains deep vanilla networks as fast as ResNets without shortcut connections.
problem Training very deep neural networks is challenging.
method Developed a new type of transformation compatible with Leaky ReLUs.
result Validation accuracies with deep vanilla networks are competitive with ResNets and significantly higher.
Study rectifying curves in 3D multiplicative Euclidean space.
problem Investigate rectifying curves in a non-Newtonian geometry setting.
method Apply multiplicative differential-geometric concepts to rectifying curves.
result Classify multiplicative rectifying curves using spherical curves.
Investigates Darboux rectifying curves on smooth surfaces.
problem Characterizing Darboux rectifying curves on smooth surfaces.
method Analyzes the position vector under isometry and finds conformal invariance conditions.
result Identifies sufficient conditions for conformal invariance of Darboux rectifying curves.
Study characterizes k-rectifiable sets in homogeneous groups.
problem Characterizing k-rectifiable sets in arbitrary homogeneous groups. method Proves characterizations using (k,G)-approximate tangent groups. result Existence of (k,G)-approximate tangent groups implies k-rectifiability. New curves generalize helix and rectifying curves.
problem Generalizing helix and rectifying curves.
method Introducing f-rectifying curves with f-position vector in rectifying plane.
result Classification and characterization of f-rectifying curves.
PHP connects to ReLU neural networks for scalable Bayesian inference.
problem Scalability and Bayesian inference in two-layer ReLU neural networks.
method PHP with Gaussian prior, decomposition propositions, annealed sequential Monte Carlo.
result PHP provides an alternative scalable representation for two-layer ReLU neural networks.
Recalls and refines the concept of algebraically rectifiable curves.
problem Classical notion of algebraically rectifiable plane curves.
method Provides new criteria, relates to quadratic differentials, and generalizes to higher order differentials.
result Generalization and new criteria for algebraic rectifiability.