S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
An appropriate choice of the activation function (like ReLU, sigmoid or swish) plays an important role in the performance of (deep) multilayer perceptrons (MLP) for classification and regression learning. Prototype-based classification learning methods like (generalized) learning vector quantization (GLVQ) are powerful…
A new KAN variant uses sinusoidal activations to approximate functions.
problem Approximating multivariable functions using neural networks.
method Replacing inner and outer functions in Kolmogorov-Arnold representation with weighted sinusoidal functions.
result The new KAN variant outperforms fixed-frequency Fourier transform and achieves comparable performance to MLPs.
In this paper we propose a synergistic melting of neural networks and decision trees (DT) we call neural decision trees (NDT). NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more…
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of …
Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
MLP-Mixer achieves better performance through sparsity and wider architecture.
problem Understanding why MLP-Mixer outperforms conventional MLPs.
method Revealed sparseness as a key mechanism, showed effective expression as wider MLP, demonstrated quantitative similarities, and applied a guiding principle.
result MLP-Mixer's performance improvement through sparseness and wider architecture.
Wav-KAN improves neural network interpretability and performance.
problem Challenges in interpretability, training speed, robustness, and performance of traditional neural networks.
method Integrates wavelet functions into the Kolmogorov-Arnold network structure for efficient data representation.
result Enhanced accuracy, faster training speeds, and increased robustness compared to existing methods.
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
PAC-Bayesian bounds for MLPs with cross entropy loss validated.
problem Generalization bounds for MLPs with cross entropy loss.
method Introduced probabilistic explanations and proved PAC-Bayesian bounds using ELBO.
result MLPs with cross entropy loss inherently guarantee PAC-Bayesian generalization bounds.
Interpolated-MLPs control inductive bias for better performance in low-compute tasks.
problem Low-compute performance gap between MLPs and CNNs.
method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.
result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
ES-MLP combines Graph-MLP with edge splitting for node classification on both homophilic and heterophilic graphs.
problem Node classification on graphs with mixed homophilic and heterophilic properties.
method Combines Graph-MLP with edge splitting mechanism from ES-GNN to learn two adjacency matrices based on relevant and irrelevant feature pairs.
result ES-MLP achieves performance comparable to homophilic and heterophilic models without using edges during inference.
A new GNM model outperforms MLP for tabular data.
problem Learning with tabular data.
method Proposes Graph Neural Machine (GNM) replacing MLP's graph representation with a nearly complete graph and using synchronous message passing.
result GNM outperforms MLP in classification and regression tasks.
This study compares GNNs and GA-MLPs, finding GA-MLPs can distinguish graphs but not count walks.
problem Comparing expressive power and graph isomorphism testing capabilities of GNNs and GA-MLPs.
method GA-MLPs augment node features with multi-hop operators and apply MLPs node-wise; GNNs are compared as a baseline.
result GA-MLPs can distinguish almost all non-isomorphic graphs but cannot count attributed walks, unlike GNNs.
Bilinear MLPs offer a new way to interpret deep learning models without complex nonlinearities.
problem Lack of mechanistic understanding in how MLPs compute.
method Introduced bilinear MLPs without element-wise nonlinearities, analyzed their weights using tensor and eigendecomposition.
result Bilinear MLPs provide interpretable weight structures and enable adversarial attacks and overfitting analysis.
KANs replace fixed MLP weights with learnable edge functions, improving accuracy and interpretability.
problem Lack of interpretability and scalability in MLPs.
method KANs use learnable activation functions on edges instead of fixed weights, replacing weights with spline functions.
result KANs outperform MLPs in accuracy and interpretability with smaller models.
Artificial Neural Networks(ANN) has been phenomenally successful on various pattern recognition tasks. However, the design of neural networks rely heavily on the experience and intuitions of individual developers. In this article, the author introduces a mathematical structure called MLP algebra on the set of all Multi…
Revisits neural collaborative filtering vs. matrix factorization, showing dot product superiority.
problem Comparing neural collaborative filtering to matrix factorization in recommendation systems.
method Revisited experiments using MLPs as similarity functions, comparing dot product to MLP outputs.
result Simple dot product outperforms MLP-based learned similarities in practical settings.
Study uses MLP models to predict large-cap US stocks, finding 2-3 hidden layers more flexible.
problem Predicting asset prices for large-cap US stocks.
method Applied MLP models with dynamic structure to factor models, focusing on firm characteristics.
result MLP models with 2-3 hidden layers more flexible in modeling factors, better for downside risk control.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios…
SpArX creates faithful explanations of neural networks' decision-making.
problem Challenges in explaining neural networks' decisions.
method Sparsifies MLPs while maintaining structure, then translates into QAFs for argumentative explanations.
result SpArX provides more faithful explanations than existing methods.
This paper proposes a new approach to Transformers by integrating hierarchical associative memory with MetaFormers.
problem Theoretical framework for Transformers and MLP-Mixers remains underdeveloped.
method Integrating hierarchical associative memory with MetaFormers to create a parallelized MLP-Mixer.
result Symmetry-breaking effects improve the performance of the MLP-Mixer in image recognition tasks.
Designs an MLP from LDA for multi-Gaussian class classification.
problem Classifying inputs with multiple Gaussian distributions.
method Interprets MLP as generalized LDA, using LDAs for half-space partitioning, neurons for subspace isolation, and merging for class-wise representation.
result Automatic feedforward design for MLP architecture and weights.
We develop a new approach to learn the parameters of regression models with hidden variables. In a nutshell, we estimate the gradient of the regression function at a set of random points, and cluster the estimated gradients. The centers of the clusters are used as estimates for the parameters of hidden units. We justif…
m-arcsinh improves SVM and MLP reliability and speed in scikit-learn.
problem Improving SVM and MLP reliability and speed in scikit-learn.
method Modified arcsinh function for kernel and activation in SVM and MLP.
result Competitive classification performance and reliability of SVM and MLP with m-arcsinh.
Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing m…
This study compares MLPs and KANs in low-data regimes, finding MLPs with personalized activation functions outperform KANs.
problem Comparing MLPs and KANs in low-data regimes.
method Introduced an effective technique for designing MLPs with unique, parameterized activation functions for each neuron.
result MLPs with personalized activation functions achieve significantly higher predictive accuracy with only a modest increase in parameters, especially in low-data regimes.
Deep conditional generative models are developed to simultaneously learn the temporal dependencies of multiple sequences. The model is designed by introducing a three-way weight tensor to capture the multiplicative interactions between side information and sequences. The proposed model builds on the Temporal Sigmoid Be…
New approach finds minimum width for deep, narrow MLPs.
problem Finding the minimum width for deep, narrow MLPs to approximate continuous functions.
method Proposes a framework to simplify finding minimum width into determining a geometrical function w(dx,dy) based on input and output dimensions. result Proves that w(dx,dy) equals the optimal minimum width for deep, narrow MLPs to achieve universality. Investigates transfer learning in spatial statistics.
problem Applying transfer learning to spatial statistics.
method Simple MLP models for spatial data.
result Potential of transfer learning in spatial statistics.
Closed-form polynomial approximations replace MLPs in transformers, enabling new interpretability methods.
problem Replacing MLPs with polynomial approximations for transformer models.
method Theoretical derivation of closed-form least-squares approximations of MLPs and GLUs using polynomial functions.
result Polynomial approximations explain over 95% of MLP and GLU outputs' variance, enabling interpretability.
ResNets promote smoother interpolations than MLPs, enhancing generalization.
problem Understanding the difference in smoothness between ResNets and MLPs.
method Neural Tangent Kernel (NTK) analysis during gradient descent training.
result ResNet's NTK results in smoother interpolations than MLPs.
Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherit…
We propose a differentiable sigmoid function for efficient p-value calculation in clustering.
problem Efficient and accurate p-value calculation for clustering algorithms.
method Designed a differentiable sigmoid function to approximate the Dip-p-value transformation.
result Accelerates computation and integrates well with gradient descent-based learning schemes.
Topo-MLP learns network representations without message passing.
problem Lack of efficient higher-order network modeling methods.
method Proposes Topo-MLP, a simplicial neural network algorithm using MLP and HONC loss.
result Demonstrates improved robustness and efficiency in representation learning.
Wind power as a renewable source of energy, has numerous economic, environmental and social benefits. In order to enhance and control renewable wind power, it is vital to utilize models that predict wind speed with high accuracy. Due to neglecting of requirement and significance of data preprocessing and disregarding t…
Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers o…
Group invariant and equivariant Multilayer Perceptrons (MLP), also known as Equivariant Networks, have achieved remarkable success in learning on a variety of data structures, such as sequences, images, sets, and graphs. Using tools from group theory, this paper proves the universality of a broad class of equivariant M…
MLPs can approximate any function in context, challenging the importance of in-context universality.
problem Understanding why transformers are more effective than classical models.
method Proved MLPs with trainable activation functions are universal in context.
result Transformer success is likely due to factors other than in-context universality.
Survival trees exhibit end-cut preference, leading to biased splits.
problem End-cut preference in survival trees causes biased splits and poor interpretability.
method Proposed a smooth sigmoid surrogate (SSS) approach to replace hard-threshold indicator function.
result Smooth sigmoid surrogate (SSS) effectively mitigates end-cut preference in survival trees.
Randomized methods of neural network learning suffer from a problem with the generation of random parameters as they are difficult to set optimally to obtain a good projection space. The standard method draws the parameters from a fixed interval which is independent of the data scope and activation function type. This …
Transformers show better in-context learning resilience under distribution shifts than simple MLPs.
problem Understanding in-context learning under varying distribution shifts.
method Comparing transformers and set-based MLPs on linear regression tasks.
result Transformers better emulate OLS performance and exhibit better resilience to mild distribution shifts.
Transformers with MLP heads outperform linear baselines in in-context learning.
problem Understanding in-context learning in Transformers with nonlinear MLP heads.
method Analyzing a model with two-layer MLPs trained via gradient steps and fully optimized, under high-dimensional asymptotics.
result Nonlinear MLPs enhance ICL performance, particularly on nonlinear tasks.