On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

9 results for “MLP-Mixers”

Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breakingcs.LG

This paper proposes a new approach to Transformers by integrating hierarchical associative memory with MetaFormers.

problem Theoretical framework for Transformers and MLP-Mixers remains underdeveloped.

method Integrating hierarchical associative memory with MetaFormers to create a parallelized MLP-Mixer.

result Symmetry-breaking effects improve the performance of the MLP-Mixer in image recognition tasks.

Transformers MLP-Mixers Hierarchical Associative Memory Symmetry Breaking Attention Mechanisms

2024-06-18abs ↗pdf ↗

Understanding MLP-Mixer as a Wide and Sparse MLPcs.LG

MLP-Mixer achieves better performance through sparsity and wider architecture.

problem Understanding why MLP-Mixer outperforms conventional MLPs.

method Revealed sparseness as a key mechanism, showed effective expression as wider MLP, demonstrated quantitative similarities, and applied a guiding principle.

result MLP-Mixer's performance improvement through sparseness and wider architecture.

MLP-Mixer sparsity wider MLP Kronecker-product weights sparse regularization

2023-06-02abs ↗pdf ↗

MixerFlow: MLP-Mixer meets Normalising Flowsstat.ML

MixerFlow combines MLP-Mixer with normalizing flows for efficient image modeling.

problem Efficiently modeling complex image densities using generative models.

method Proposes MixerFlow, a novel architecture based on MLP-Mixer for normalizing flows.

result Demonstrates improved density estimation and better scaling with higher image resolutions.

normalizing flows MLP-Mixer image modeling generative models density estimation

2023-10-25abs ↗pdf ↗

Interpolated-MLPs: Controllable Inductive Biascs.LG

Interpolated-MLPs control inductive bias for better performance in low-compute tasks.

problem Low-compute performance gap between MLPs and CNNs.

method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.

result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.

Inductive Bias Interpolated MLP Low-Compute Vision Tasks CNN

2024-10-12abs ↗pdf ↗

Do Vision Transformers See Like Convolutional Neural Networks?cs.CV

Vision Transformers show different internal representations compared to CNNs.

problem Understanding how Vision Transformers solve image classification tasks.

method Comparative analysis of ViT and CNN architectures on image classification benchmarks.

result ViT has more uniform representations across all layers, while CNNs have more varied representations.

Vision Transformers Convolutional Neural Networks Image Classification Self-Attention Residual Connections

2021-08-19abs ↗pdf ↗

Exploring the Limits of Large Scale Pre-trainingcs.LG

Study shows scaling up models doesn't always improve downstream tasks.

problem Understanding why scaling up models doesn't always improve downstream performance.

method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.

result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.

model scaling downstream performance saturation representation evolution nonlinear relationship

2021-10-05abs ↗pdf ↗

Bayesian sparsification for deep neural networks with Bayesian model reductionstat.ML

Bayesian sparsification reduces deep neural network complexity.

problem Complexity of deep neural networks limits their performance.

method Combines Bayesian shrinkage priors with stochastic variational inference.

result Bayesian model reduction (BMR) is a more efficient alternative for pruning model weights.

Bayesian sparsification deep learning model reduction stochastic variational inference Bayesian model reduction

2023-09-21abs ↗pdf ↗

Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscs.LG

New method diagnoses criticality in deep neural networks, improving performance.

problem Improving theoretical understanding and practical initialization of deep neural networks.

method Introducing partial Jacobians and deriving recurrence relations for their norms to analyze criticality.

result Proper stacking of LayerNorm and residual connections leads to a critical architecture for any initialization.

deep learning neural networks criticality partial Jacobians LayerNorm

2021-11-23abs ↗pdf ↗

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscs.LG

Transformers exhibit sparse activation maps, reducing computational load and improving robustness.

problem Sparse activation in Transformer models.

method Extensive experiments on various Transformer architectures and tasks.

result Sparsity in Transformers is a prevalent phenomenon, reducing FLOP count and improving model robustness.

Transformer Sparse activation Machine learning Efficiency Robustness

2022-10-12abs ↗pdf ↗