Papers/2611.17339
🧪 Test?View on arXiv

Does the Ranking Hold? A Practitioner-Facing Audit of Hallucination Detectors on Small, Quantized, On-Device LLMs

Author1, Author2, Author3, Author4, Author5

hallucination detectionquantized modelson-device AIevaluation metrics
2611.17339
Builder Relevance
70%
19h ago

Abstract

This paper audits hallucination detectors on small, quantized LLMs, revealing poor cross-model ranking and methodological issues with correctness labeling.

Reality Card

Core Claim

The study demonstrates that the ranking of hallucination detectors is inconsistent across different small models, with a significant inversion in performance rankings.

Method / Result

Cross-model ranking transfer is poor: Spearman rho is -0.80 and Kendall tau-b is -0.667.

Limitations

The standard token-overlap correctness labeler produces near-chance AUROC scores, indicating a failure in labeling rather than detector performance.

Paper to code

Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.

No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.
← Back to all papers