When Should Forecasting Agents Reason? Behavioral Stress Tests for Reliability Routing
Louis Wang, Author 2, Author 3, Author 4, Author 5
Abstract
The paper investigates the conditions under which different forecasting agent behaviors should be trusted, revealing that the choice of mechanism is source-dependent.
Reality Card
The study introduces ReliabilityRoute, a structural intervention that optimizes forecasting-agent behavior based on reliability features, achieving a competitive mean Brier score across various LLM vintages.
A walk-forward self-adjusting rule achieved the best mean Brier score among deterministic systems across 16 later LLM vintages.
Historical/search baselines remain highly competitive, indicating that the improvements may not be substantial.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.