Group invariant and equivariant Multilayer Perceptrons (MLP), also known as Equivariant Networks, have achieved remarkable success in learning on a variety of data structures, such as sequences, images, sets, and graphs. Using tools from group theory, this paper proves the universality of a broad class of equivariant M…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “MLP layers”
This study compares GNNs and GA-MLPs, finding GA-MLPs can distinguish graphs but not count walks.
problem Comparing expressive power and graph isomorphism testing capabilities of GNNs and GA-MLPs.
method GA-MLPs augment node features with multi-hop operators and apply MLPs node-wise; GNNs are compared as a baseline.
result GA-MLPs can distinguish almost all non-isomorphic graphs but cannot count attributed walks, unlike GNNs.
Designs an MLP from LDA for multi-Gaussian class classification.
problem Classifying inputs with multiple Gaussian distributions.
method Interprets MLP as generalized LDA, using LDAs for half-space partitioning, neurons for subspace isolation, and merging for class-wise representation.
result Automatic feedforward design for MLP architecture and weights.
Paper hypothesizes MLP layers in LLMs can be approximated by sparse Mixture of Experts.
problem Understanding dense MLP layers in LLMs.
method Theoretical connection between MoE models and SAE structure in activation space.
result MLP layers in LLMs can be well approximated by sparse Mixture of Experts.
MLP-Mixer achieves better performance through sparsity and wider architecture.
problem Understanding why MLP-Mixer outperforms conventional MLPs.
method Revealed sparseness as a key mechanism, showed effective expression as wider MLP, demonstrated quantitative similarities, and applied a guiding principle.
result MLP-Mixer's performance improvement through sparseness and wider architecture.
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Bilinear MLPs offer a new way to interpret deep learning models without complex nonlinearities.
problem Lack of mechanistic understanding in how MLPs compute.
method Introduced bilinear MLPs without element-wise nonlinearities, analyzed their weights using tensor and eigendecomposition.
result Bilinear MLPs provide interpretable weight structures and enable adversarial attacks and overfitting analysis.
This paper proposes a new approach to Transformers by integrating hierarchical associative memory with MetaFormers.
problem Theoretical framework for Transformers and MLP-Mixers remains underdeveloped.
method Integrating hierarchical associative memory with MetaFormers to create a parallelized MLP-Mixer.
result Symmetry-breaking effects improve the performance of the MLP-Mixer in image recognition tasks.
Study uses MLP models to predict large-cap US stocks, finding 2-3 hidden layers more flexible.
problem Predicting asset prices for large-cap US stocks.
method Applied MLP models with dynamic structure to factor models, focusing on firm characteristics.
result MLP models with 2-3 hidden layers more flexible in modeling factors, better for downside risk control.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
problem Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
method Formal study of PAC learnability of MLP layers with activation sparsity.
result Classes of functions with activation sparsity lead to provable computational and statistical advantages over their non-sparse counterparts.
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPsstat.ML
Transformers with MLP heads outperform linear baselines in in-context learning.
problem Understanding in-context learning in Transformers with nonlinear MLP heads.
method Analyzing a model with two-layer MLPs trained via gradient steps and fully optimized, under high-dimensional asymptotics.
result Nonlinear MLPs enhance ICL performance, particularly on nonlinear tasks.
This paper tackles overfitting in CTR models by introducing Multi-Epoch learning with Data Augmentation.
problem Overfitting of the embedding layer in CTR models during multi-epoch training.
method Introduces Multi-Epoch learning with Data Augmentation (MEDA) framework to reduce overfitting and enhance performance.
result MEDA minimizes overfitting and achieves data augmentation through varied embedding spaces, improving performance without overfitting.
A new GNM model outperforms MLP for tabular data.
problem Learning with tabular data.
method Proposes Graph Neural Machine (GNM) replacing MLP's graph representation with a nearly complete graph and using synchronous message passing.
result GNM outperforms MLP in classification and regression tasks.
Transformers exhibit sparse activation maps, reducing computational load and improving robustness.
problem Sparse activation in Transformer models.
method Extensive experiments on various Transformer architectures and tasks.
result Sparsity in Transformers is a prevalent phenomenon, reducing FLOP count and improving model robustness.
A new perspective on self-attention models using MLPs.
problem Improving sequence modeling with self-attention mechanisms.
method Introducing HyperMLP and HyperGLU, which use dynamic two-layer MLPs with reverse-offset layout.
result HyperMLP/HyperGLU consistently outperform softmax-attention baselines.
Interpolated-MLPs control inductive bias for better performance in low-compute tasks.
problem Low-compute performance gap between MLPs and CNNs.
method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.
result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.
KANs replace fixed MLP weights with learnable edge functions, improving accuracy and interpretability.
problem Lack of interpretability and scalability in MLPs.
method KANs use learnable activation functions on edges instead of fixed weights, replacing weights with spline functions.
result KANs outperform MLPs in accuracy and interpretability with smaller models.
A new transformer model accelerates training with optimization techniques.
problem Training deep neural networks efficiently and effectively.
method Interprets transformer layers as optimization steps, applying Nesterov acceleration.
result The new model outperforms existing models on benchmark datasets.
Linear Transformer Block combines MLP and linear attention for near-optimal ICL in linear regression.
problem Achieving near-optimal in-context learning (ICL) risk for linear regression with a Gaussian prior.
method Combines linear attention and MLP components in a Linear Transformer Block (LTB). Establishes correspondence with one-step gradient descent estimators ().
result LTB achieves nearly Bayes optimal ICL risk for linear regression with a Gaussian prior.
Wind power as a renewable source of energy, has numerous economic, environmental and social benefits. In order to enhance and control renewable wind power, it is vital to utilize models that predict wind speed with high accuracy. Due to neglecting of requirement and significance of data preprocessing and disregarding t…
In the analysis of machine learning models, it is often convenient to assume that the parameters are IID. This assumption is not satisfied when the parameters are updated through training processes such as SGD. A relaxation of the IID condition is a probabilistic symmetry known as exchangeability. We show the sense in …
SpArX creates faithful explanations of neural networks' decision-making.
problem Challenges in explaining neural networks' decisions.
method Sparsifies MLPs while maintaining structure, then translates into QAFs for argumentative explanations.
result SpArX provides more faithful explanations than existing methods.
Proposes a balanced multi-component and multi-layer neural network for efficient function approximation.
problem Accurately and efficiently approximating complex functions with high degrees of freedom and computational cost.
method Inspired by a multi-component approach, MMNN combines single-layer networks with a multi-layer decomposition strategy.
result Significant reduction in training parameters, more efficient training process, and improved accuracy compared to FCNNs or MLPs.
This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
problem Proving the asymptotic freeness of layerwise Jacobians in multilayer perceptrons (MLPs).
method Replacing each layer's parameter matrix with itself multiplied by a Haar orthogonal matrix, and using the invariance of the MLP.
result Proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
m-arcsinh improves SVM and MLP reliability and speed in scikit-learn.
problem Improving SVM and MLP reliability and speed in scikit-learn.
method Modified arcsinh function for kernel and activation in SVM and MLP.
result Competitive classification performance and reliability of SVM and MLP with m-arcsinh.
Deep neural networks with heavy-tailed weights converge to stable distributions.
problem Understanding the convergence of heavy-tailed weights in infinitely-wide neural networks.
method Analyzing infinitely-wide multi-layer perceptrons with i.i.d. symmetric -stable weight distributions.
result The vector of pre-activation values converges to i.i.d. symmetric -stable distributions.
A new model improves the performance of knowledge distillation in GNNs.
problem Inconsistent performance of existing knowledge distillation techniques in GNNs.
method Proposes a new model, Routing-by-Memory (RbM), a form of Mixture-of-Experts (MoE), to address performance concerns.
result Demonstrates experimentally that RbM achieves considerably more consistent performance across multiple datasets.
HaKAN uses Hahn-KAN blocks to forecast multivariate time series.
problem Long-term time series forecasting challenges with high complexity and spectral bias.
method HaKAN integrates channel independence, patching, and a stack of Hahn-KAN blocks with residual connections. It uses Hahn polynomial-based learnable activation functions.
result HaKAN consistently outperforms state-of-the-art methods on various forecasting benchmarks.
We propose a new type of hidden layer for a multilayer perceptron, and demonstrate that it obtains the best reported performance for an MLP on the MNIST dataset.
Transformers without skip connections collapse token representations to a single direction.
problem Rapid convergence of token representations to a single direction in self-attention-only Transformers.
method Analysis of layer normalization, residual connections, and multi-head attention mechanisms.
result Residual connections prevent rank collapse in real Transformers, while MLPs generate new feature directions.
New method initializes MLPs for tabular data with tree-based feature interactions.
problem No dedicated neural network architecture for tabular data.
method Sparse tree-based initialization for MLPs.
result New method outperforms default MLP initialization and gradient boosting.
This work concerns testing the number of parameters in one hidden layer multilayer perceptron (MLP). For this purpose we assume that we have identifiable models, up to a finite group of transformations on the weights, this is for example the case when the number of hidden units is know. In this framework, we show that …
New methods use Kronecker-factored approximations for faster deep learning optimization.
problem Optimizing deep learning models with rich curvature information.
method Approximate Hessian using Kronecker products for efficient quasi-Newton methods.
result New methods outperform first-order methods and perform comparably to second-order methods.
Random Transformers behave like polynomial models in ICL with asymptotic growth.
problem Understanding in-context learning capabilities of pretrained Transformers.
method Asymptotic analysis of a random Transformer with a fixed first layer and a trained second layer, considering growth in context length, input dimension, hidden dimension, and training parameters.
result The random Transformer's ICL error is equivalent to a finite-degree Hermite polynomial model.
We propose Very Simple Classifier (VSC) a novel method designed to incorporate the concepts of subsampling and locality in the definition of features to be used as the input of a perceptron. The rationale is that locality theoretically guarantees a bound on the generalization error. Each feature in VSC is a max-margin …
Neural Decision Treesstat.ML
In this paper we propose a synergistic melting of neural networks and decision trees (DT) we call neural decision trees (NDT). NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more…
Proof of learning rate transfer in MLPs with P parameterization.
problem Understanding and optimizing learning rates in neural networks with different parameterizations.
method Theoretical analysis and empirical validation of learning rate transfer in MLPs with P, SP, and NTP parameterizations.
result The optimal learning rate converges to a non-zero constant as width goes to infinity under P, explaining learning rate transfer.
Transformers show better in-context learning resilience under distribution shifts than simple MLPs.
problem Understanding in-context learning under varying distribution shifts.
method Comparing transformers and set-based MLPs on linear regression tasks.
result Transformers better emulate OLS performance and exhibit better resilience to mild distribution shifts.
New approach finds minimum width for deep, narrow MLPs.
problem Finding the minimum width for deep, narrow MLPs to approximate continuous functions.
method Proposes a framework to simplify finding minimum width into determining a geometrical function based on input and output dimensions.
result Proves that equals the optimal minimum width for deep, narrow MLPs to achieve universality.
Transformers can scale both context and task, but MLPs can only scale task.
problem Understanding and scaling In-Context Learning in transformers.
method Simplified transformer architecture, feature map, and MLP combination.
result Simplified transformer can perform ICL and context-scaling but not task-scaling.
An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of…
Estimates MLP expected output without sampling, using fewer FLOPs.
problem Estimating expected loss in machine learning efficiently.
method Approximate representations of activations using cumulants and Hermite expansions.
result Estimates expected output with fewer FLOPs than Monte Carlo sampling.
TiDE uses MLP for fast, simple long-term time-series forecasting.
problem Long-term time-series forecasting challenges.
method Time-series Dense Encoder (TiDE) based on MLP.
result TiDE matches or outperforms Transformer models while being 5-10x faster.
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
Deep neural networks can grok better than shallow ones, showing multi-stage generalization.
problem Understanding the generalization behavior of deep neural networks.
method Empirical replication and analysis of grokking phenomenon in deep MLP models.
result Deep neural networks exhibit multi-stage generalization, with a secondary surge in test accuracy.