This work interprets GELU and related activations via a first-order loss function.
problem Understanding and optimizing activation functions in neural networks.
method Complementary interpretation using the Gaussian first-order loss function.
result Calibrated or learned uniform-threshold gates are competitive and often outperform GELU, ReLU, and SiLU/Swish.
New kernels from ELU and GELU networks reveal non-trivial fixed points.
problem Understanding fixed-point dynamics in deep neural networks with ELU and GELU activations.
method Deriving covariance functions and analyzing fixed-point dynamics of ELU and GELU networks.
result ELU and GELU networks exhibit non-trivial fixed-point dynamics, explaining implicit regularization in overparameterized models.
In this paper, a novel neural network activation function, called Symmetrical Gaussian Error Linear Unit (SGELU), is proposed to obtain high performance. It is achieved by effectively integrating the property of the stochastic regularizer in the Gaussian Error Linear Unit (GELU) with the symmetrical characteristics. Co…
Randomly initialized wide neural networks with zero-mean activations are nearly independent, potentially solving AI interpretability limits.
problem Measuring the limits of AI interpretability.
method Randomly initialized neural networks with large width and zero-mean activation functions.
result Neural networks with zero-mean activations are nearly independent, solving the computational no-coincidence conjecture.
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.
GLU variants improve Transformer performance.
problem Improving the performance of Transformer models.
method Tested variations of Gated Linear Units (GLU) in Transformer feed-forward sublayers.
result Some GLU variants yield better performance than ReLU or GELU activations.
New bounds for shallow neural networks with deterministic parameters.
problem Developing generalisation bounds for shallow neural networks.
method PAC-Bayesian theory applied to shallow neural networks with deterministic parameters.
result Empirical non-vacuous bounds for shallow neural networks trained with vanilla SGD.
Paper presents a method to accurately quantify neural network uncertainty without sampling.
problem Uncertainty quantification in neural networks for reliability and robustness.
method Sample-free moment propagation technique for mean vectors and covariance matrices.
result Analytic solution for covariance of nonlinear activation functions.
We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning. K-TanH consists of parameterized low-precision integer operations, such as, shift and add/subtract (no floating point operation needed) where parameters are stored in very small look-up t…
Study learns a neuron with non-monotonic activation functions.
problem Learning a single neuron with non-monotonic activation functions.
method Gradient descent (GD) with conditions on activation function and input distribution.
result Learnability of non-monotonic activation functions is established without monotonicity assumption.
Gradient flows of neural networks converge to optimal values or diverge, with thresholds and asymptotic behaviors.
problem Understanding the convergence and divergence of gradient flows in neural networks.
method Analysis of gradient flows on loss landscapes of neural networks using o-minimal structures.
result Gradient flows either converge to optimal values or diverge to infinity, with thresholds and asymptotic behaviors.
Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.
problem The issue of oversmoothing in attention-based GNNs.
method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.
result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.
Improved robustness for deep neural networks with tighter bounds and attacks.
problem Loose upper bounds and prohibitive computation in existing adversarial robustness methods.
method Primal approach with exact Lipschitz certificates for ReLU networks and modern architectures, and novel Wasserstein Distributional Attacks.
result Tighter upper bounds and greater flexibility in attack points compared to existing methods.
Lower bound proves ridgeless regression performs poorly near interpolation threshold.
problem Proving performance of ridgeless regression near interpolation threshold.
method Distribution-independent lower bound for mean squared error in noisy ridgeless linear regression.
result Lower bound implies ridgeless regression performs poorly near interpolation threshold.
Quantized neural networks can represent all fixed-point functions under certain conditions.
problem Expressive power of quantized neural networks under fixed-point arithmetic.
method Analyzing necessary and sufficient conditions for quantized networks to represent all fixed-point functions.
result Various popular activation functions satisfy the sufficient condition for representing all fixed-point functions.
Minimum width for ReLU networks on compact domain is exactly max{d_x, d_y, 2}
problem Characterizing the minimum width for ReLU networks to approximate functions on compact domains
method Analyzing the minimum width for Lp approximation of Lp functions from [0,1]d to Rdy using ReLU-like activation functions result The minimum width for Lp approximation on a compact domain is exactly max{d_x, d_y, 2} for ReLU-like activation functions Token-adaptive FFN design improves LLM expressivity.
problem Fixed activation functions limit FFN expressivity in LLMs.
method Mixture of Activations (MoA) and learnable activations (LA).
result MoA achieves lower terminal loss and better scaling than baselines.
This paper precisely estimates transformer derivatives for explicit learning guarantees.
problem Computing fully-explicit generalization bounds for transformers with precise higher-order derivative estimates.
method Analyzes and estimates all higher-order derivatives of transformers with multiple attention heads and layer normalization.
result Obtains explicit pathwise generalization bounds for transformers learning from non-i.i.d. samples.
Unified framework proves neural networks' ability to mimic complex tasks.
problem Lack of a single constructive framework for neural network universality.
method Introduces neural network approximate identity (nAI) and proves it leads to universality.
result Any nAI activation function is universal.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
Weibull framework diagnoses transformer weight distributions, revealing distinct patterns across modules.
problem Diagnosing weight distributions in transformer models for better understanding of training dynamics.
method Applied Weibull distribution to diagnose weight magnitude distributions in transformer models, fitting each weight matrix independently.
result Distinct patterns of weight distributions across different transformer modules were identified.