Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
Author1, Author2, Author3, Author4, Author5
Abstract
This paper explores the efficiency of using a frozen vocabulary in small language models, demonstrating that it can outperform traditional learned embeddings in certain contexts.
Reality Card
The frozen embedding table of 16 MB significantly reduces model size and improves performance on rare words while maintaining competitive results on various benchmarks.
The frozen-table model generates tokens at a rate of 615 to 740 tokens per second on a laptop CPU, achieving 5 to 6 times the speed of the float model.
The study's findings may not generalize across all language tasks, as performance varies by benchmark.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.