🧪 Test?View on arXiv
Does the Ranking Hold? A Practitioner-Facing Audit of Hallucination Detectors on Small, Quantized, On-Device LLMs
Author1, Author2, Author3, Author4, Author5
hallucination detectionquantized modelson-device AIevaluation metrics
2611.17339
Builder Relevance
19h ago70%
Abstract
This paper audits hallucination detectors on small, quantized LLMs, revealing poor cross-model ranking and methodological issues with correctness labeling.
Reality Card
Core Claim
The study demonstrates that the ranking of hallucination detectors is inconsistent across different small models, with a significant inversion in performance rankings.
Method / Result
Cross-model ranking transfer is poor: Spearman rho is -0.80 and Kendall tau-b is -0.667.
Limitations
The standard token-overlap correctness labeler produces near-chance AUROC scores, indicating a failure in labeling rather than detector performance.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.
No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.