AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks
Not specified in the abstract
Abstract
AlphaDiana is a measurement system designed to analyze agent performance in reasoning tasks by linking execution conditions to outcomes and resource use.
Reality Card
AlphaDiana demonstrates that harnesses do not consistently improve reasoning accuracy and that component interventions like skill guidance and memory retention do not reliably enhance performance.
The study found that gains on unsolved questions can be outweighed by losses on questions answered correctly without tools.
The inconsistency in performance improvements raises concerns about the reliability of harnesses and interventions in enhancing agent accuracy.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.