🧪 Test?View on arXiv
FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law
Athanasios Zeris
attention mechanismslanguage modelingfilter design
2610.00009
Builder Relevance
20h ago80%
Abstract
This paper investigates the effectiveness of frequency-collapse attention using bandpass-filtered inner products, revealing optimal filter properties for improved performance in language modeling.
Reality Card
Core Claim
The optimal single-scale bandwidth for frequency-collapse attention is approximately 2 bins centered at paragraph scale, achieving a clean gain of +1.15 nats over standard dot-product attention.
Method / Result
Optimal bandwidth of sigma ~= 2 bins leads to a performance improvement of Delta = +1.15 nats.
Limitations
The findings are primarily based on character-level language modeling, which may not directly translate to other contexts or models.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.
No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.