The Drift Contract: Spectral Updates for Depth-Robust Local Learning
Not specified in the provided content
Abstract
The paper presents a novel approach to local learning that addresses accuracy degradation and hyperparameter fragility in deep networks by applying spectral update geometry.
Reality Card
The spectral update method significantly improves depth robustness in local learning, outperforming local Adam optimizers with a consistent step-size setting across various depths and widths.
The spectral update achieved an accuracy of 48.9 +/- 0.5 compared to 46.6 +/- 0.3 for local Adam at width 512.
The stability benefit of spectral updates is not observed when using RMSNorm and weight decay, limiting its effectiveness in certain configurations.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.