Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
Not provided in the abstract
Abstract
This paper investigates the transition from memorization to generalization in neural networks, specifically focusing on the phenomenon known as grokking.
Reality Card
The study quantitatively characterizes the memorization-to-generalization boundary in hyperparameter space, revealing that data complexity is the dominant factor influencing the transition.
The power-law scaling relation for generalization onset time is given by: T_grok ∝ H^{-0.27} D^{-2.04} η^{-0.50} λ^{-0.64}, with R^2 = 0.732.
The study focuses on a specific architecture (two-hidden-layer MLPs) and modular arithmetic, which may limit generalizability to other architectures or tasks.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.