On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

2 results for “Active-Attention”

Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attentionstat.ML

Paper connects MoE and self-attention, proposing active-attention.

problem Improving efficiency and performance of self-attention mechanisms.

method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.

result Active-attention outperforms standard self-attention in various tasks.

Mixture of Experts Self-Attention Quadratic Gating Active-Attention Efficiency

2024-10-15abs ↗pdf ↗

The Quarks of Attentioncs.LG

Investigates the fundamental components of attention mechanisms.

problem Understanding the building blocks of attention in deep learning.

method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.

result Additive activation attention is central in proofs of lower bounds.

attention mechanisms deep learning transformer architectures multiplicative mechanisms gating mechanisms

2022-02-15abs ↗pdf ↗