VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages
Not provided in the content
Abstract
The paper introduces VakyArth, a pragmatic benchmark for Indic languages, highlighting the challenges faced by multilingual LLMs in understanding pragmatic meanings rooted in Indic linguistic and cultural conventions.
Reality Card
VakyArth reveals systematic failures in multilingual LLMs' pragmatic understanding across Indic languages, with significant differences in performance based on language and task type.
MCQ accuracy exceeds NLI accuracy in all model-language combinations.
The study's findings may not be generalizable beyond the specific Indic languages tested, and automatic translation metrics may overlook pragmatic fidelity.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.