🧪 Test?View on arXiv
Evaluation protocols for historical script classification: the cost of manuscript-level leakage and the role of soft labels
Not provided in the abstract
classificationsoft labelsevaluation protocolshistorical manuscripts
2611.12969
Builder Relevance
8h ago70%
Abstract
This paper evaluates the impact of manuscript-level leakage on script classification accuracy and the effectiveness of soft labels in improving model predictions.
Reality Card
Core Claim
The study demonstrates that testing on pages from training manuscripts significantly inflates accuracy metrics, with a notable drop in page-level accuracy from 97% to 54%.
Method / Result
Page-level accuracy drops from 97% to 54%, and the expected calibration error (ECE) rises significantly.
Limitations
The accuracy difference with soft labels could not be confirmed or ruled out due to the limited number of manuscripts.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.
No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.