We study deep Bayesian neural networks with Gaussian priors, revealing heavy-tailed unit activations.
problem Characterizing regularization effects in deep Bayesian neural networks.
method Investigation of deep Bayesian neural networks with Gaussian weight priors and ReLU-like nonlinearities.
result The prior distribution on units becomes increasingly heavy-tailed with depth, influencing activation patterns.
ReLU activations lead to smoother learning curves compared to sigmoidal activations in neural networks.
problem Comparing the performance of ReLU and sigmoidal activations in neural networks.
method Analytical computation of learning curves in shallow networks with different activation functions.
result ReLU networks exhibit continuous transitions in performance, while sigmoidal networks show discontinuous transitions.
Neural networks approximate unit spheres as polytopes.
problem Approximating unit spheres with neural networks.
method Using ReLU activation in neural networks to generate polytopes.
result Neural networks can approximate unit spheres as polytopes.
In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evalu…
DEUs learn nonlinear activation functions from data, reducing network size.
problem Fixed activation functions in neural networks limit performance.
method Differential equation units (DEUs) learn nonlinear activation functions from data.
result DEUs enable neurons to change their activation functions during training.
Paper proposes a new activation function to reduce overfitting and large weight update issues.
problem Overfitting and large weight update problems in neural networks.
method Introduces a new activation function called Thresholded Exponential Rectified Linear Units (TERELU).
result TERELU shows better performance in reducing overfitting and large weight update issues compared to other activation functions.
SPLASH units improve deep neural network accuracy and adversarial robustness.
problem Improving deep neural network accuracy and adversarial robustness.
method SPLASH units are learnable activation functions with specific properties that enhance both accuracy and robustness.
result SPLASH units significantly increase adversarial robustness of deep neural networks.
ReLU units can 'die' in neural networks, causing slower convergence.
problem ReLU units sometimes produce near-zero outputs during training.
method Simulation and statistical analysis of a simplified ReLU unit model.
result Activation probability decreases as training progresses, leading to slower convergence.
A new method for the unsupervised learning of sparse representations using autoencoders is proposed and implemented by ordering the output of the hidden units by their activation value and progressively reconstructing the input in this order. This can be done efficiently in parallel with the use of cumulative sums and …
The most widely used activation functions in current deep feed-forward neural networks are rectified linear units (ReLU), and many alternatives have been successfully applied, as well. However, none of the alternatives have managed to consistently outperform the rest and there is no unified theory connecting properties…
Improved perceptron design mitigates vanishing gradient problem.
problem Vanishing gradient problem in deep multilayer perceptrons.
method Auto-rotating perceptron (ARP) design with dynamic activation region.
result Neural networks with ARP units achieve better learning performance.
AReLU uses attention-based rectification to improve neural network performance.
problem Improving neural network performance through better activation functions.
method Integrates attention mechanism with rectified linear unit (ReLU) to learn and scale feature maps.
result AReLU significantly boosts performance of most network architectures with minimal changes.
New neural network uses differential equations for adaptable activation functions.
problem Fixed activation functions limit neural network performance and size.
method Introduces differential equation units (DEUs) that learn nonlinear activation functions.
result DEUs enable more compact networks with comparable performance.
Improved activation function NLReLU boosts neural network performance.
problem Performance issues with ReLU activation function.
method NLReLU uses parametric natural logarithmic transform to improve ReLU.
result NLReLU provides higher accuracy than ReLU in various neural networks.
Methodology selects best activation functions for DNN layers.
problem Challenging to choose the best activation function for each DNN layer.
method Identifies Evaluation Points and dropout rates for each layer.
result Average 7% to 15% Relative Error Reduction on benchmarks.
Algorithm learns two-layer residual units using ReLU activations from samples.
problem Learning two-layer residual units from samples.
method Design layer-wise objectives as functionals, formulate ERM as QP, solve using LP, prove statistical consistency.
result Strong statistical consistency and robustness of the algorithm.
CNNs classify human activities from IMU data.
problem Automatic identification of physical activities using motion sensors.
method Used Convolutional Neural Networks (CNNs) with raw IMU data.
result CNNs perform well in classifying 16 lower-limb activities.
Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot …
We establish upper bounds for the minimal number of hidden units for which a binary stochastic feedforward network with sigmoid activation probabilities and a single hidden layer is a universal approximator of Markov kernels. We show that each possible probabilistic assignment of the states of n output units, given t…
Proposes PER loss to regularize neural network activations to normal distribution.
problem Improving neural network generalization and training speed.
method Regularizes activations to standard normal distribution via projected error function and Wasserstein distance.
result Minimizes Wasserstein distance between activation distribution and standard normal.
Neural networks struggle with identity relations; DR units improve generalization.
problem Neural networks fail to generalize identity relations.
method Exploring various factors in neural network architecture and learning process, including number of hidden layers, activation function, and data representation.
result DR units improve generalization, leading to almost perfect test accuracy in mid fusion setting.
New activations improve deep network reproducibility without sacrificing accuracy.
problem Deep networks' reproducibility issues, especially on distributed systems.
method Developed SmeLU activations, smoother than ReLU, to enhance reproducibility.
result SmeLU activations provide better accuracy-reproducibility tradeoffs.
A new recurrent unit alleviates vanishing gradients for long-term dependencies.
problem Vanishing gradients in recurrent neural networks make long-term dependencies hard to model.
method Proposes a new NRU architecture that avoids saturating activation functions and gates.
result Demonstrates superior performance across various tasks with and without long-term dependencies.
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences.…
GOLS finds activation functions affect training robustness, especially ReLU.
problem Investigate how different activation functions impact GOLS in neural network training.
method Identify SNN-GPPs for GOLS, analyze activation function effects on gradient continuity.
result GOLS robust for most activation functions but sensitive to ReLU.
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural ne…
`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical advancement of machine learning. However in the field of deep learning, they have been largely displaced by rectified linear units (ReLU) or similar functions, such as its exponential linear unit (ELU) v…
New method learns neural network activation functions from data.
problem Learning activation functions for neural networks.
method Model each neuron's activation function as a small neural network.
result Learned activation functions improve network performance.
Paper derives a fast learning rate for deep neural networks without scale invariant activation functions.
problem Analyzing the impact of non-scale invariant activation functions on deep learning performance.
method Using Suzuki (2018) framework, derived a tight generalization error bound for deep neural networks with non-scale invariant activations.
result Without scale invariance of activation functions, deep learning can still achieve a fast learning rate.
Deep gated networks help understand training and generalization in deep learning.
problem Understanding the role of SGD in training and generalization of deep neural networks with ReLU activation.
method Developed deep gated networks (DGNs) as a framework to analyze training and generalization in DNNs with ReLU activation.
result Gate adaptation is key for generalization in deep neural networks.
Automates feature extraction for IMU-based activity recognition.
problem Manual feature engineering is time-consuming and limits IMU-based applications.
method FRESH algorithm for automated feature extraction.
result Workflow automates feature extraction from synchronized IMU sensors.
Dropout training improves neural networks' performance.
problem Improving neural network convergence and generalization.
method Two-layer neural networks with ReLU activations, overparametrization, and positive margin assumption.
result Dropout training achieves ε-suboptimality in test error in O(1/ε) iterations.
We study the activity, i.e., the number of transactions per unit time, of financial markets. Using the diffusion entropy technique we show that the autocorrelation of the activity is caused by the presence of peaks whose time distances are distributed following an asymptotic power law which ultimately recovers the Pois…
Study tail risk in high-frequency finance using L1-regularized regression.
problem Measuring tail risk dynamics in high-frequency financial markets.
method Dynamic extreme value regression model with L1-regularized maximum likelihood estimator. result Severity of extreme losses well predicted by low price impact in high volatility periods.
This work tackles representation learning by introducing stochastic competition-based activations.
problem Learning diversified representations in deep learning models.
method Combining information-theoretic arguments with stochastic competition-based activations, using Stochastic Local Winner-Takes-All (LWTA) units.
result The proposed method yields significant discriminative representation learning abilities and allows for a principled investigation of intermediate network representations.
Improves neural network performance by normalizing activation functions.
problem Improving convergence speed and robustness of neural networks.
method Transforming existing activation functions into ones with better properties.
result Significantly promotes convergence robustness, maximum training depth, and anytime performance.
Activation functions have a notorious impact on neural networks on both training and testing the models against the desired problem. Currently, the most used activation function is the Rectified Linear Unit (ReLU). This paper introduces a new and novel activation function, closely related with the new activation $Swish…
Model neural plasticity as binary optimization to dynamically activate or deactivate network units.
problem Dynamic learning and adaptability of neural networks.
method Model neural plasticity as an L0-norm regularized binary optimization problem, where units can be activated or deactivated based on a cost-benefit tradeoff. result Demonstrates that a single parameter k can modulate learning dynamics, unifying network sparsification and expansion. This paper explores transforming deep neural networks into simpler structures while preserving or approximating their functionality.
problem Transforming deep neural networks into simpler structures while preserving or approximating their functionality.
method Constructive proof of global linear approximation, removal of inactive or always active ReLUs, and experiments with regularization and adversarial training.
result For any feed-forward ReLU network, there exists a global linear approximation to a 2-hidden-layer shallow network with a fixed number of units.
Evolutionary algorithms improve neural network performance by discovering better activation functions.
problem The choice of activation function affects neural network performance, but ReLU remains dominant.
method Defined a tree-based search space of candidate activation functions and used evolutionary algorithms (mutation, crossover, exhaustive search) to explore and discover better functions.
result Replacing ReLU with evolved activation functions statistically significantly increases network accuracy.
Proposes SGELU for neural networks to improve performance and convergence.
problem Improving neural network performance and convergence.
method Integrates Gaussian Error Linear Unit (GELU) with symmetrical characteristics to create SGELU.
result SGELU outperforms GELU and LiSHT in MNIST classification and auto-encoder tasks.
Study shows semi-supervised learning improves human activity recognition with minimal user input.
problem Improving human activity recognition models using incremental learning.
method Three approaches: non-supervised, semi-supervised, and supervised learning were compared.
result Semi-supervised learning achieves similar accuracy to supervised learning with minimal user input.
modAL is a modular active learning framework for Python, aimed to make active learning research and practice simpler. Its distinguishing features are (i) clear and modular object oriented design (ii) full compatibility with scikit-learn models and workflows. These features make fast prototyping and easy extensibility p…
Study embeddings between Barron spaces with various activation functions, focusing on RePU.
problem Understanding the influence of activation functions on infinitely wide neural networks.
method Prove embeddings by constructing push-forward maps on measures representing functions.
result Barron spaces with RePU activation have a hierarchical structure similar to Sobolev spaces.
Identifies learning rules from neural network observables.
problem Determine the underlying plasticity rules governing learning in biological systems.
method Simulated idealized neuroscience experiments with artificial neural networks to generate a dataset of learning trajectories. Used linear and non-linear classifiers to identify learning rules from aggregate statistics of weights, activations, and activity changes.
result Different classes of learning rules can be separated solely on the basis of aggregate statistics of the weights, activations, or instantaneous layer-wise activity changes.
Foreign exchange markets show that currency units (= accounting or nominal price units) are variables. Technical and economic progress evidence that the consumer baskets (= purchasing power units or real price units) are also variables. In contrast, all physical measurement units are constants and either defined in the…
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. Th…
Automatically discovers effective activation functions for deep learning.
problem Inconsistent performance of novel activation functions in deep learning networks.
method Evolutionary search for general form, gradient descent for parameters.
result Significant performance improvements over ReLU and other functions.