Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs
Author1, Author2, Author3, Author4, Author5
Abstract
The study investigates the amount of synthetic data required for activation probes monitoring language models, emphasizing that coverage is more critical than difficulty in determining sample needs.
Reality Card
The number of synthetic samples needed for activation probes is determined by coverage rather than the difficulty of the data, with a median half-gain size of 7-11 samples across different concepts.
Probes for high-stakes and harmful situations plateau with as few as 80 samples, while instruction probes require several times more.
The variance in results is significantly influenced by the concept and distribution, which may affect reproducibility across different contexts.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.