Papers/2611.24963
🧪 Test?View on arXiv

Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

Not provided in the content

architecturepretrainingreasoningbenchmarking
2611.24963
Builder Relevance
70%
2d ago

Abstract

The paper investigates architectural differences in Transformers and linear models during pretraining, revealing that Canon layers improve performance while challenging traditional benchmark-based architecture research.

Reality Card

Core Claim

Canon layers consistently improve Transformers, saving about 30% of pretraining tokens for the same MMLU accuracy at 8B parameters.

Method / Result

At 1-2T tokens, Canon layers improve performance while linear models can match or exceed Transformers on benchmarks but struggle with basic in-context skills.

Limitations

High scores on standard benchmarks can obscure weaknesses in basic in-context skills, and no model reliably beats random guessing on simple two-hop questions.

Paper to code

Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.

No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.
← Back to all papers