🧪 Test?View on arXiv
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Not provided
annotationexpert feedbackLLMcodebook revision
2609.26926
Builder Relevance
2h ago80%
Abstract
This paper explores how large language models can expedite the codebook revision process for large-scale text annotation by identifying areas of disagreement that require expert input.
Reality Card
Core Claim
LLMs can strategically target expert attention, reducing codebook revision time from months to days while improving labeling performance.
Method / Result
Rationale Labeling achieved 64.9% LLM-labeling accuracy against expert labels, outperforming the expert-revised codebook accuracy of 57.8%.
Limitations
The study may have limitations in generalizability due to the specific dataset used (tutoring-session transcripts).
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.
No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.