New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
Study on learning dynamics in deep neural networks, proving properties and confirming empirical observations.
problem Understanding the learning dynamics of deep neural networks, especially in binary classification.
method Proved properties of learning in binary classification under strong assumptions, extended existing results from linear case.
result Classification error follows a sigmoidal shape in nonlinear architectures, parallel independent modes of learning, gradient starvation phenomenon.
Develops ADMM for deep neural networks with sigmoid activations to avoid saturation and improve approximation.
problem Gradient saturation in deep neural networks with sigmoid activations.
method Introduces sigmoid-ADMM pair for training deep sigmoid nets and proves its convergence.
result ADMM avoids saturation and improves approximation of deep sigmoid nets compared to ReLU nets.
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
The highly detailed international trade data among all countries in the world during 1971-2000 shows that the kinds of export goods and the logarithmic GDP (gross domestic production) of a country has an S-shaped relationship. This indicates all countries can be divided into three stages accordingly. First, the poor co…
Paper proposes HTAF for stable training of binary neural networks.
problem Challenges in training binary neural networks with gradient-based optimization.
method HTAF is a smooth approximation to the Heaviside function that enables stable training.
result HTAF enables stable training of various binary neural networks with gradient-based optimization.
New bounds for shallow neural networks with deterministic parameters.
problem Developing generalisation bounds for shallow neural networks.
method PAC-Bayesian theory applied to shallow neural networks with deterministic parameters.
result Empirical non-vacuous bounds for shallow neural networks trained with vanilla SGD.
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
ReLU activations lead to smoother learning curves compared to sigmoidal activations in neural networks.
problem Comparing the performance of ReLU and sigmoidal activations in neural networks.
method Analytical computation of learning curves in shallow networks with different activation functions.
result ReLU networks exhibit continuous transitions in performance, while sigmoidal networks show discontinuous transitions.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Proposes a new LSTM gate structure using bivariate Beta distribution.
problem Inflexibility of sigmoid gates in modeling multi-modality and skewness, and lack of modeling correlation between gates.
method Introduces a bivariate Beta distribution gate structure within LSTM cells.
result Empirically shows higher gradient values and improved model performance.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
New method optimizes neural network learning by adjusting random parameters to target function features.
problem Difficulty in setting optimal random parameters for neural network learning.
method Adjusts sigmoid slopes and positions to target function features in a randomized learning method.
result Significantly better approximation of complex target functions compared to standard methods.
We develop a new approach to learn the parameters of regression models with hidden variables. In a nutshell, we estimate the gradient of the regression function at a set of random points, and cluster the estimated gradients. The centers of the clusters are used as estimates for the parameters of hidden units. We justif…
Shallow neural nets classify objects perfectly if their distribution is linearly separable.
problem Designing efficient neural networks for classification.
method Constructed shallow sigmoid-type neural networks.
result Achieves 100% accuracy for datasets following a linear separability condition.
Deep conditional generative models are developed to simultaneously learn the temporal dependencies of multiple sequences. The model is designed by introducing a three-way weight tensor to capture the multiplicative interactions between side information and sequences. The proposed model builds on the Temporal Sigmoid Be…
We simplify word embeddings by removing sigmoid in SGNS, revealing connections to hyperbolic spaces.
problem Improving word embeddings quality and understanding their relationship with hyperbolic spaces.
method Analyzing squashed shifted PMI matrix and its relation to graph properties and hyperbolic geometry.
result Word embeddings can be connected to hyperbolic spaces through squashed shifted PMI matrix.
Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherit…
Efficiently infers Poisson process intensity using Gaussian process with sigmoid link.
problem Estimating intensity of inhomogeneous Poisson processes efficiently.
method Variational free-form mean field optimization and sparse Laplace's method.
result Method is one order of magnitude faster than exact inference and competitive with quadratic link function models.
We propose a differentiable sigmoid function for efficient p-value calculation in clustering.
problem Efficient and accurate p-value calculation for clustering algorithms.
method Designed a differentiable sigmoid function to approximate the Dip-p-value transformation.
result Accelerates computation and integrates well with gradient descent-based learning schemes.
Survival trees exhibit end-cut preference, leading to biased splits.
problem End-cut preference in survival trees causes biased splits and poor interpretability.
method Proposed a smooth sigmoid surrogate (SSS) approach to replace hard-threshold indicator function.
result Smooth sigmoid surrogate (SSS) effectively mitigates end-cut preference in survival trees.
GLU variants improve Transformer performance.
problem Improving the performance of Transformer models.
method Tested variations of Gated Linear Units (GLU) in Transformer feed-forward sublayers.
result Some GLU variants yield better performance than ReLU or GELU activations.
We developed safe and ranged approximations for ReLU, tanh, and sigmoid functions to reduce neural network training time.
problem Expensive computation of hyperbolic tangent and sigmoid functions in neural networks.
method Function approximation techniques to create safe and ranged approximations.
result 10% to 37% improvement in training times on CPU and 20% to 53% improvement in ranged cases.
Researchers use statistical physics to model neural network learning dynamics.
problem Understanding the learning dynamics of ReLU neural networks.
method Developed a system of differential equations using statistical physics techniques.
result ReLU networks exhibit distinct learning behavior compared to sigmoidal networks.
Adapts neural network neurons' activation functions for better predictions.
problem Training neural networks with fixed activation functions limits their performance.
method Proposes training over a shape parameter, allowing neurons to adapt their own activation functions.
result Improves prediction accuracy by allowing neurons to tune their activation functions.
Tempered sigmoids improve deep learning privacy.
problem Privacy-preserving deep learning with strict differential privacy guarantees.
method Developed tempered sigmoid activation functions for deep learning models.
result Tempered sigmoids outperform ReLU in achieving state-of-the-art accuracy.
Python scripts analyze MRAM-based neuromorphic devices' process variation impacts on machine learning accuracy.
problem Impact of process variation on MRAM-based neuromorphic devices' performance in machine learning applications.
method Developed transportable Python scripts to analyze output variation under changes in device dimensions.
result Revealed impacts and limits for processing variation of device fabrication on energy vs. accuracy tradeoffs.
Paper develops privacy-preserving mechanisms for machine learning using wavelet transforms.
problem Improper data privacy methods compromise user data even with small preliminary knowledge.
method Three privacy-preserving mechanisms with discrete M-band wavelet transform.
result Successfully retains both differential privacy and learnability in various machine learning environments.
The study explores how Matrix Product States can represent boolean and continuous functions.
problem Representing arbitrary boolean and continuous functions using Matrix Product States.
method Developed a construction method for MPS to represent boolean gates and proved density in continuous function space.
result MPS can accurately represent arbitrary boolean functions and continuous functions densely.
New method improves random parameter generation in neural networks.
problem Standard method of generating random weights and biases in neural networks has drawbacks.
method Proposes a new method to generate random parameters ensuring nonlinear sigmoids remain in the input hypercube and uniformly distributed slope angles for activation functions.
result Ensures the most useful nonlinear fragments of sigmoids remain in the input hypercube.
Proposes a method to visualize finer cluster structures in high-dimensional data.
problem Visualization of high-dimensional data with complex cluster structures.
method Introduces a generalized sigmoid function with a parameter b to adjust the tail heaviness for better visualization.
result The method can generate visualization results comparable to UMAP, revealing finer cluster structures.
Paper investigates differentiable fuzzy implications and their suitability for learning.
problem Analyzing differentiable fuzzy implications and their suitability for learning.
method Investigates the properties of fuzzy implications in a differentiable setting and introduces a new family of fuzzy implications.
result Various fuzzy implications are unsuitable for differentiable learning, and a new family of fuzzy implications is introduced.
This note is concerned with accurate and computationally efficient approximations of moments of Gaussian random variables passed through sigmoid or softmax mappings. These approximations are semi-analytical (i.e. they involve the numerical adjustment of parametric forms) and highly accurate (they yield 5% error at most…
Balanced Activation improves object detection performance on long-tailed datasets.
problem Mismatch between training and testing label distributions in object detection.
method Introduces Balanced Activation (Balanced Softmax and Balanced Sigmoid) to address label distribution shift.
result Balanced Activation provides ~3% gain in mAP on LVIS-1.0 compared to state-of-the-art methods.
Monotonic differentiable sorting networks improve upon previous methods.
problem Non-monotonicity in differentiable sorting networks.
method Relaxation of conditional swap operations using sigmoid functions to ensure monotonicity.
result Monotonic differentiable sorting networks improve upon previous methods.
Activation functions have a notorious impact on neural networks on both training and testing the models against the desired problem. Currently, the most used activation function is the Rectified Linear Unit (ReLU). This paper introduces a new and novel activation function, closely related with the new activation $Swish…
Adaptive pricing framework for perpetual contracts using liquidity curves and oracles.
problem Ensuring stable and predictable pricing for perpetual contracts.
method Uses liquidity curves and on-chain oracles with parabolic and sigmoid functions to quote prices and fees.
result Ensures pricing stability and predictability through adaptive pricing framework.
Paper proposes algorithms for active learning of reject option classifiers.
problem Active learning of reject option classifiers is unaddressed in machine learning.
method Developed novel algorithms using double ramp and double sigmoid loss functions.
result Proposed algorithms efficiently reduce the number of labeled examples required.
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is O(1) is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condi…
Neural networks can approximate any L^p functions on R^n.
problem Approximating functions on unbounded domains with neural networks.
method Monotone sigmoid, ReLU, ELU, Softplus, LeakyReLU activation functions.
result Shallow neural networks can arbitrarily well approximate L^p functions on R^n.
SummaryNet automates video summarisation using deep learning.
problem Creating informative video summaries from videos.
method Two-stream convolutional network for spatial and temporal features, encoder-decoder model for salient features, sigmoid regression with LSTM for frame probability.
result SummaryNet achieves comparable or better results than state-of-the-art methods on benchmark datasets.
A new method sparsifies neural networks using stochastic binary optimization.
problem Sparsifying neural networks to reduce computational cost and improve efficiency.
method Stochastic binary optimization with the Augment-Reinforce-Merge (ARM) estimator.
result ARM enables efficient network sparsification with comparable accuracy to baseline methods.
We propose a new static parameterization of the implied volatility surface which is constructed by using polynomials of sigmoid functions combined with some other terms. This parameterization is flexible enough to fit market implied volatilities which demonstrate smile or skew. An arbitrage-free calibration algorithm i…
Highly expressive directed latent variable models, such as sigmoid belief networks, are difficult to train on large datasets because exact inference in them is intractable and none of the approximate inference methods that have been applied to them scale well. We propose a fast non-iterative approximate inference metho…
New neural network units resist adversarial attacks effectively.
problem Adversarial attacks on neural networks that misclassify inputs.
method Introduced RBFI units with non-linear structure.
result RBFI units maintain high accuracy in adversarial attacks.