On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

3 results for “Linformer”

Linformer: Self-Attention with Linear Complexitycs.LG

Linformer reduces transformer complexity to linear, improving efficiency.

problem High cost of training and deploying large transformer models for long sequences.

method Approximates self-attention with low-rank matrix, proposing Linformer with

O(n)

complexity.

result Linformer performs similarly to standard transformers but is more memory- and time-efficient.

transformer self-attention linear complexity natural language processing memory efficiency

2020-06-08abs ↗pdf ↗

Sumformer: Universal Approximation for Efficient Transformerscs.LG

Sumformer simplifies Transformers to handle long sequences efficiently.

problem Quadratic complexity of Transformers limits their use with long sequences.

method Introducing Sumformer, a simple architecture that universally approximates equivariant sequence-to-sequence functions.

result Sumformer achieves the first universal approximation results for Linformer and Performer.

Sumformer Transformers Universal approximation Linformer Performer

2023-07-05abs ↗pdf ↗

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequencescs.LG

Skeinformer accelerates self-attention for long sequences with linear complexity.

problem Efficiency of Transformer models in processing long sequences.

method Matrix sketching and column sampling to reduce quadratic complexity to linear.

result Skeinformer outperforms alternatives with smaller time/space footprint.

Transformer models self-attention long sequences matrix sketching quadratic complexity

2021-12-10abs ↗pdf ↗