New method improves random parameter generation in neural networks.
problem Standard method of generating random weights and biases in neural networks has drawbacks.
method Proposes a new method to generate random parameters ensuring nonlinear sigmoids remain in the input hypercube and uniformly distributed slope angles for activation functions.
result Ensures the most useful nonlinear fragments of sigmoids remain in the input hypercube.
GLU variants improve Transformer performance.
problem Improving the performance of Transformer models.
method Tested variations of Gated Linear Units (GLU) in Transformer feed-forward sublayers.
result Some GLU variants yield better performance than ReLU or GELU activations.
New method optimizes neural network learning by adjusting random parameters to target function features.
problem Difficulty in setting optimal random parameters for neural network learning.
method Adjusts sigmoid slopes and positions to target function features in a randomized learning method.
result Significantly better approximation of complex target functions compared to standard methods.
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is O(1) is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condi…
We present a probabilistic framework for nonlinearities, based on doubly truncated Gaussian distributions. By setting the truncation points appropriately, we are able to generate various types of nonlinearities within a unified framework, including sigmoid, tanh and ReLU, the most commonly used nonlinearities in neural…
Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value …
We give a polynomial-time algorithm for learning neural networks with one layer of sigmoids feeding into any Lipschitz, monotone activation function (e.g., sigmoid or ReLU). We make no assumptions on the structure of the network, and the algorithm succeeds with respect to {\em any} distribution on the unit ball in n …
We show that any smooth bi-Lipschitz h can be represented exactly as a composition hm∘...∘h1 of functions h1,...,hm that are close to the identity in the sense that each (hi−Id) is Lipschitz, and the Lipschitz constant decreases inversely with the number m of functions com…
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of …
The paper proves neural network identifiability for a broad range of nonlinearities.
problem Can a neural network's architecture, weights, and biases be uniquely determined by its input-output map?
method Derive necessary genericity conditions for identifiability of neural networks of arbitrary depth and connectivity with an arbitrary nonlinearity.
result Construct a family of nonlinearities for which these genericity conditions are minimal, necessary, and sufficient.
Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
DEUs learn nonlinear activation functions from data, reducing network size.
problem Fixed activation functions in neural networks limit performance.
method Differential equation units (DEUs) learn nonlinear activation functions from data.
result DEUs enable neurons to change their activation functions during training.
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
ReLU activations lead to smoother learning curves compared to sigmoidal activations in neural networks.
problem Comparing the performance of ReLU and sigmoidal activations in neural networks.
method Analytical computation of learning curves in shallow networks with different activation functions.
result ReLU networks exhibit continuous transitions in performance, while sigmoidal networks show discontinuous transitions.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
Develops a flexible model for regime transitions in time series data.
problem Nonlinear and context-dependent regime transitions in time series data.
method Semi-parametric state-space model with learned transition functions.
result Improved recovery of nonlinear transition dynamics and earlier detection of regime changes.
We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insigh…
We develop embeddings for nonlinear subspaces preserving vector norms.
problem Preserving vector norms in nonlinear subspaces.
method Low-distortion embeddings for subspaces under nonlinear transformations.
result First low-distortion embeddings for a wide class of nonlinear functions.
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation functio…
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Improved learning bounds for corrupted data using thresholded gradient descent.
problem Learning from corrupted data with adversarial noise.
method Thresholded gradient descent for sigmoidal, leaky-ReLU, and ReLU activations.
result Improved approximation bounds for various activation functions.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
We develop a new approach to learn the parameters of regression models with hidden variables. In a nutshell, we estimate the gradient of the regression function at a set of random points, and cluster the estimated gradients. The centers of the clusters are used as estimates for the parameters of hidden units. We justif…
Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing m…
Deep conditional generative models are developed to simultaneously learn the temporal dependencies of multiple sequences. The model is designed by introducing a three-way weight tensor to capture the multiplicative interactions between side information and sequences. The proposed model builds on the Temporal Sigmoid Be…
We simplify word embeddings by removing sigmoid in SGNS, revealing connections to hyperbolic spaces.
problem Improving word embeddings quality and understanding their relationship with hyperbolic spaces.
method Analyzing squashed shifted PMI matrix and its relation to graph properties and hyperbolic geometry.
result Word embeddings can be connected to hyperbolic spaces through squashed shifted PMI matrix.
A complex-valued convolutional network (convnet) implements the repeated application of the following composition of three operations, recursively applying the composition to an input vector of nonnegative real numbers: (1) convolution with complex-valued vectors followed by (2) taking the absolute value of every entry…
Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherit…
We propose a differentiable sigmoid function for efficient p-value calculation in clustering.
problem Efficient and accurate p-value calculation for clustering algorithms.
method Designed a differentiable sigmoid function to approximate the Dip-p-value transformation.
result Accelerates computation and integrates well with gradient descent-based learning schemes.
A new KAN variant uses sinusoidal activations to approximate functions.
problem Approximating multivariable functions using neural networks.
method Replacing inner and outer functions in Kolmogorov-Arnold representation with weighted sinusoidal functions.
result The new KAN variant outperforms fixed-frequency Fourier transform and achieves comparable performance to MLPs.
Bayesian Perceptron offers fully Bayesian neural networks without complex computations.
problem Lack of uncertainty quantification in neural networks.
method Bayesian inference framework for perceptron training and predictions in closed-form.
result Analytical expressions for perceptron's output and weight learning provided.
New scalable variational Bayes methods for Hawkes processes.
problem Computational intractability of Bayesian estimation for generalised nonlinear Hawkes processes.
method Unified variational Bayes framework, adaptive mean-field approximation, sparsity-inducing procedure.
result Adaptive mean-field variational algorithm for sigmoid Hawkes processes is scalable and robust.
Survival trees exhibit end-cut preference, leading to biased splits.
problem End-cut preference in survival trees causes biased splits and poor interpretability.
method Proposed a smooth sigmoid surrogate (SSS) approach to replace hard-threshold indicator function.
result Smooth sigmoid surrogate (SSS) effectively mitigates end-cut preference in survival trees.
While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood. In this work, we study the case of binary classification and prove various properties of learning in such networks under strong assumptions such as linear separabil…
We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched …
We developed safe and ranged approximations for ReLU, tanh, and sigmoid functions to reduce neural network training time.
problem Expensive computation of hyperbolic tangent and sigmoid functions in neural networks.
method Function approximation techniques to create safe and ranged approximations.
result 10% to 37% improvement in training times on CPU and 20% to 53% improvement in ranged cases.
Guiding the design of neural networks is of great importance to save enormous resources consumed on empirical decisions of architectural parameters. This paper constructs shallow sigmoid-type neural networks that achieve 100% accuracy in classification for datasets following a linear separability condition. The separab…
The goal of a recommendation system is to predict the interest of a user in a given item by exploiting the existing set of ratings as well as certain user/item features. A standard approach to modeling this problem is Inductive Matrix Completion where the predicted rating is modeled as an inner product of the user and …
This paper explores adaptive neural activation in RNNs for better learning.
problem Fixed neural activation functions limit the performance and adaptability of RNNs.
method Developed a novel parametric family of nonlinear activation functions inspired by biological neurons.
result Adaptive neural activation improves learning speed and performance in RNNs.
Large-scale recurrent networks have drawn increasing attention recently because of their capabilities in modeling a large variety of real-world phenomena and physical mechanisms. This paper studies how to identify all authentic connections and estimate system parameters of a recurrent network, given a sequence of node …
Tempered sigmoids improve deep learning privacy.
problem Privacy-preserving deep learning with strict differential privacy guarantees.
method Developed tempered sigmoid activation functions for deep learning models.
result Tempered sigmoids outperform ReLU in achieving state-of-the-art accuracy.
Python scripts analyze MRAM-based neuromorphic devices' process variation impacts on machine learning accuracy.
problem Impact of process variation on MRAM-based neuromorphic devices' performance in machine learning applications.
method Developed transportable Python scripts to analyze output variation under changes in device dimensions.
result Revealed impacts and limits for processing variation of device fabrication on energy vs. accuracy tradeoffs.
Paper develops privacy-preserving mechanisms for machine learning using wavelet transforms.
problem Improper data privacy methods compromise user data even with small preliminary knowledge.
method Three privacy-preserving mechanisms with discrete M-band wavelet transform.
result Successfully retains both differential privacy and learnability in various machine learning environments.
The study explores how Matrix Product States can represent boolean and continuous functions.
problem Representing arbitrary boolean and continuous functions using Matrix Product States.
method Developed a construction method for MPS to represent boolean gates and proved density in continuous function space.
result MPS can accurately represent arbitrary boolean functions and continuous functions densely.
DIFFRNN verifies RNNs for safety and equivalence.
problem Certifying safety and equivalence of RNNs for resource-constrained platforms.
method Bounding nonlinear activation functions with linear constraints, solving constrained optimization problems, and using dReal SMT solver.
result DIFFRNN certifies the equivalence of optimized and original RNNs.