This paper explores adaptive neural activation in RNNs for better learning.
problem Fixed neural activation functions limit the performance and adaptability of RNNs.
method Developed a novel parametric family of nonlinear activation functions inspired by biological neurons.
result Adaptive neural activation improves learning speed and performance in RNNs.
Deep learning models outperform classical methods in forecasting neural activity.
problem Improving forecasting of neural activity using deep learning models.
method Systematic evaluation of eight probabilistic deep learning models against classical statistical models and baseline methods.
result Several deep learning models consistently outperform classical approaches in forecasting neural activity.
This paper provides an overview of activation functions in neural networks.
problem Confusion in activation function selection and properties in deep learning.
method Analytic review of popular activation functions.
result Clarification of activation function properties and selection.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Large deviation principle for deep neural networks with ReLU activation.
problem Understanding the behavior of deep neural networks with ReLU activation.
method Proving a large deviation principle for networks with Gaussian weights and ReLU activation functions.
result Simplified expressions and power-series expansions for the ReLU case.
We present a comprehensive study of multilayer neural networks with binary activation, relying on the PAC-Bayesian theory. Our contributions are twofold: (i) we develop an end-to-end framework to train a binary activated deep neural network, (ii) we provide nonvacuous PAC-Bayesian generalization bounds for binary activ…
There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functi…
Study challenges the Gaussian pre-activations assumption in neural networks.
problem Challenges the assumption that pre-activations are Gaussian in neural networks.
method Constructs pairs of activation functions and initialization distributions to ensure Gaussian pre-activations.
result Discovered constraints for ensuring Gaussian pre-activations in neural networks.
Neural networks learn distance-based representations, not just intensity.
problem Understanding how neural networks interpret and learn from internal activations.
method Manipulated ReLU and Absolute Value activations to observe sensitivity to distance and intensity perturbations.
result Neural networks are highly sensitive to small distance-based perturbations, challenging the intensity-based interpretation.
Data-aware activation function customization reduces neural network error.
problem Current neural networks lack consideration for specific activation functions.
method Linear algebraic explanation and Diaconis-Shahshahani Approximation Theorem criteria for activation functions.
result Using an even activation function like seagull can reduce neural network error by orders of magnitude.
Periodic activation functions improve neural network reliability and interpretability.
problem Neural networks reinforce hidden biases, making them unreliable and hard to interpret.
method Introduce periodic activation functions in Bayesian neural networks to establish a connection with stationary Gaussian process priors.
result Periodic activation functions, including sinusoidal, triangular, and ReLU, improve model performance and sensitivity to perturbations.
Abstract: Investigates the role of activation functions in neural networks and their physical basis.
problem Understanding the role of activation functions in neural networks and their physical basis.
method Formalizes the use of activation functions in neural inference by relating them to phase transitions in statistical physics.
result Reveals the physical justification for the performance of typical activation functions in neural networks.
Unified study of active learning for deep neural networks.
problem Improving performance of deep neural networks through active learning.
method Investigation of incremental and cumulative training modes, model configurations, query strategies, and pseudo-labels.
result Proposed more efficient querying procedures and insights into active learning behavior.
Polynomial neural networks explore thresholds for maximum expressiveness.
problem Understanding the limits of polynomial neural networks' expressiveness.
method Introducing activation degree threshold to measure network expressiveness and proving its existence and upper bounds.
result Polynomial neural networks with equi-width architectures achieve the maximum expressiveness.
Many neural network architectures rely on the choice of the activation function for each hidden layer. Given the activation function, the neural network is trained over the bias and the weight parameters. The bias catches the center of the activation, and the weights capture the scale. Here we propose to train the netw…
New method speeds up neural kernel computations for various activations.
problem Inefficient computation of neural kernels for general activations.
method Fast sketching method using truncated Hermite expansion.
result 106x speedup for approximate CNTK computation on CIFAR-10.
Rational neural networks approximate functions more efficiently with less depth.
problem Choosing optimal nonlinear activation functions in neural networks.
method Rational activation functions with optimal bounds and efficiency proofs.
result Rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth.
An artificial neuron is modelled as a weighted summation followed by an activation function which determines its output. A wide variety of activation functions such as rectified linear units (ReLU), leaky-ReLU, Swish, MISH, etc. have been explored in the literature. In this short paper, we explore what happens when the…
Many activation functions have been proposed in the past, but selecting an adequate one requires trial and error. We propose a new methodology of designing activation functions within a neural network at each layer. We call this technique an "activation ensemble" because it allows the use of multiple activation functio…
Study shows how activation functions impact the storage capacity of treelike neural networks.
problem Understanding the role of activation functions in neural network expressive power.
method Analysis of treelike two-layer networks with various activation functions in the infinite-width limit.
result Activation functions affect storage capacity and robustness, with nonlinearity increasing capacity and decreasing robustness.
Paper examines power consumption in neural networks using various activation functions.
problem Power consumption in machine learning models.
method Examines power consumption for different activation functions.
result Substantial differences in power consumption exist between activation functions.
New method trains neural networks with threshold activation functions efficiently.
problem Training neural networks with threshold activation functions is challenging due to zero gradients.
method We study weight decay regularized training problems of deep neural networks with threshold activations, showing they can be formulated as convex optimization problems.
result Regularized deep threshold network training problems can be formulated as standard convex optimization problems, paralleling the LASSO method.
The paper analyzes the role of ReLU gates in deep learning networks.
problem Understanding the role of gates in deep learning networks.
method Developed neural path features (NPF) and neural path values (NPV) to characterize the active sub-networks during training.
result The neural path kernel associated with NPFs is a fundamental quantity that characterizes the information stored in the gates of a DNN.
The paper proves deep neural networks with analytic activation can approximate any function.
problem Approximating functions with neural networks using analytic activation functions.
method Elementary proofs for real and complex networks, Stone-Weierstrass theorem, Mergelyan's theorem.
result Closure of neural network classes equals space of polynomials for analytic activation.
Survey of trainable activation functions in neural networks.
problem Improving neural network performance through trainable activation functions.
method Taxonomy and comparison of recent and past models of trainable activation functions.
result Many trainable activation functions are equivalent to adding neuron layers with fixed activation functions and simple constraints.
Leaky ReLU activations improve the calibration of Bayesian neural networks.
problem Bayesian neural networks struggle with mean-field variational inference for ReLU activations.
method Investigated the effect of activation functions on the calibration of Bayesian neural networks.
result Leaky ReLU activations lead to more Gaussian-like weight posteriors and lower expected calibration error.
Develops wavelet-based neural network approximation theory.
problem Analyzing neural network approximation capabilities over various activation functions.
method Wavelet frame theory on spaces of homogeneous type, sufficient conditions for approximation, error estimates.
result Derives sufficient conditions for neural networks to approximate any functions in a given space, including non-smooth activations.
Active learning method for neural population dynamics using optogenetics.
problem Efficiently selecting neurons to stimulate for identifying neural population dynamics.
method Developed active learning procedure for low-rank regression to determine informative photostimulation patterns.
result Demonstrated a two-fold reduction in data required for predictive power using low-rank linear dynamical systems model.
LCW reduces activation shift in neural networks, improving training efficiency and generalization.
problem Activation shift in neural networks leading to non-zero mean preactivation values.
method Linearly constrained weights (LCW) to reduce activation shift in fully connected and convolutional layers.
result LCW resolves the vanishing gradient problem and improves generalization of neural networks.
Paper provides label complexity guarantees for deep active learning.
problem Lack of rigorous label complexity guarantees for deep active learning.
method Studied deep active learning from nonparametric classification perspective.
result Proved near-optimal label complexity guarantees for deep active learning.
Deep neural networks with piecewise-polynomial activations can approximate smooth functions and their derivatives.
problem Approximating smooth functions and their derivatives with neural networks.
method Derives the depth, width, and sparsity required for approximation in Hölder norms.
result Deep neural networks with bounded weights can approximate Hölder smooth functions and their derivatives.
Complexity measures for neural nets with general activations using path-based norms.
problem Control complexity of neural networks with arbitrary activation functions.
method Approximate general activations with ReLU networks and derive path-based norms for complexity control.
result Preliminary analyses of function spaces and regularized estimators.
The study proves a quantitative functional CLT for neural networks with smooth activation functions.
problem Understanding the convergence rates of neural networks with different activation functions.
method Functional versions of the Stein-Malliavin approach and a quantitative functional central limit theorem.
result Rates of convergence depend on the smoothness of the activation function, ranging from logarithmic to sqrt(n).
The study examines if ReLU activation function is optimal for modularity in neural networks.
problem Finding the best activation function for modularity in neural networks.
method Comparing ReLU with other activation functions for modularity and performance.
result ReLU may not be the best choice for modularity, suggesting other functions could be more suitable.
Overwhelming theoretical and empirical evidence shows that mildly overparametrized neural networks -- those with more connections than the size of the training data -- are often able to memorize the training data with 100% accuracy. This was rigorously proved for networks with sigmoid activation functions and, very …
The paper investigates how activation functions impact the training of Neural ODEs, leading to global convergence.
problem Challenges in training Neural ODEs, particularly gradient computation accuracy and convergence analysis.
method Investigates the impact of activation functions on the training dynamics of Neural ODEs.
result Establishes global convergence of Neural ODEs under gradient descent in overparameterized regimes.
Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
problem Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
method Formal study of PAC learnability of MLP layers with activation sparsity.
result Classes of functions with activation sparsity lead to provable computational and statistical advantages over their non-sparse counterparts.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
Paper proposes LANN to measure model complexity of neural networks with curve activation functions.
problem Measuring model complexity of neural networks with curve activation functions.
method Proposes LANN, a piecewise linear framework to approximate curve activation functions, and derives complexity measure based on the number of linear regions.
result Demonstrates positive correlation between overfitting and model complexity during training.
New activation functions achieve arbitrary-accuracy Sobolev approximation by fixed-size neural networks.
problem Approximation of Sobolev functions by neural networks
method Elementary Universal Activation Function and Differentiable Universal Activation Functions
result Arbitrary-accuracy Sobolev approximation by fixed-size neural networks
Unified framework proves neural networks' ability to mimic complex tasks.
problem Lack of a single constructive framework for neural network universality.
method Introduces neural network approximate identity (nAI) and proves it leads to universality.
result Any nAI activation function is universal.
Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, w…
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural ne…
Mixtures of neural operators reduce active complexity in operator learning.
problem Reduction of active complexity in operator learning models.
method Constructive comparison between routed mixtures of neural operators (MoNOs) and a fixed single-neural-operator construction.
result Every scalar uniformly continuous nonlinear operator can be approximated by a MoNO whose active expert has smaller depth, width, and rank scaling.
Neural networks struggle with periodic functions, a new activation fixes this.
problem Neural networks fail to learn simple periodic functions.
method Proposed a new activation function, x+sin2(x), to learn periodic functions. result The new activation function successfully learns and predicts periodic functions.
Smooth activations enable optimal error rates in neural networks for Sobolev function classes.
problem Achieving optimal approximation and estimation error rates for neural networks in Sobolev function classes.
method Study of neural networks with smooth activations, proving optimal rates via approximation and statistical properties.
result Constant-depth networks with smooth activations achieve optimal rates of approximation and estimation, demonstrating smoothness adaptivity.
The representations learned by deep neural networks are difficult to interpret in part due to their large parameter space and the complexities introduced by their multi-layer structure. We introduce a method for computing persistent homology over the graphical activation structure of neural networks, which provides acc…