🧪 Test?View on arXiv
Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks
Not provided in the content
benchmarkingpolicy evaluationauditdecision-making
2611.07042
Builder Relevance
3h ago80%
Abstract
The paper audits public policy-based decision benchmarks and reveals significant errors that affect scoring and decision-making.
Reality Card
Core Claim
Correcting scoring errors in benchmarks can significantly improve acceptance rates of model responses, with an increase of over 80 percentage points in some cases.
Method / Result
On the 96 LegalBench SARA numeric cases, correcting the scorer raised acceptance by more than 80 percentage points for each of three models.
Limitations
The study relies on specific benchmarks that may not generalize to all policy-based decision systems.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.
No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.