Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantization
P. Varshney, A. Author, B. Author, C. Author, D. Author
Abstract
The paper argues that reported statistics like cosine similarity are insufficient for interpreting the preservation of interpretability artifacts when transitioning from full-precision to quantized weights in AI models.
Reality Card
The study demonstrates that cosine similarity cannot be reliably interpreted without knowing the underlying class separation and sample size, leading to potential misinterpretations of model performance under quantization.
At INT4 quantization, the direction of the decision variable rotated significantly, exceeding the estimator's noise, while at INT8, no movement was detected.
The main limitation is the lack of reported sample sizes in existing studies, which hinders the ability to assess the reliability of cosine similarity as a measure of preservation.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.