BenchMIRT: New Method for Auditing LLM Benchmarks by AllenAI

BenchMIRT is a new method introduced by AllenAI for auditing large language model (LLM) benchmarks at the level of individual prompts. It utilizes multidimensional Item Response Theory (MIRT) to separate different capabilities measured by benchmarks, revealing insights into how models perform on various tasks.