🧪 Test?View on arXiv
Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States
Author1, Author2, Author3, Author4, Author5
transformerslinear-timeattention mechanismsrecurrent states
2610.20637
Builder Relevance
1h ago80%
Abstract
This paper presents a method to achieve linear-time transformers by expanding the attention kernel into a Taylor series, resulting in a fixed number of recurrent states.
Reality Card
Core Claim
The authors achieved a linear-time transformer model that maintains accuracy while reducing per-token decoding latency significantly compared to traditional softmax methods.
Method / Result
Per-token decoding latency is constant at 7.3 µs, achieving a 76.0× speedup at N = 65536.
Limitations
The requirement for query and key normalization may limit the generalizability of the method.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.
No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.