When the Preconditioning Exponent Turns Negative: Learning-Rate Coupling and Cross-Environment Generalization
Author1, Author2, Author3, Author4, Author5
Abstract
This study investigates the interaction between learning rate and preconditioning exponent in adaptive optimizers, revealing that the optimal exponent for cross-environment accuracy decreases with the logarithm of the learning rate.
Reality Card
The exponent maximizing cross-environment accuracy decreases almost linearly with the logarithm of the learning rate, indicating a complex relationship between learning rate and preconditioning in adaptive optimizers.
The fitted slopes of the relationship between exponent and learning rate range from -0.270 to -0.300, with R^2 values between 0.972 and 0.996.
The study is based on a single-seed, finite-budget mechanism study, which may limit the generalizability of the findings.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.