MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring
Not specified in the provided content
Abstract
This paper investigates the effectiveness of large language models as math tutors and the reliability of their evaluation metrics.
Reality Card
The study reveals a significant discrepancy between benchmark-style exact-match scoring and human-calibrated evaluations, indicating that current evaluation methods may not accurately reflect LLM tutoring effectiveness.
The measured Remediation Success Rate was 42.1% according to benchmark scoring, while human-calibrated evaluations reported 81.3%, showing a 39.2 percentage-point spread.
The study's results are based on simulated dialogues rather than real student interactions, raising concerns about the applicability of findings to actual learning scenarios.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.