arXiv:2605.27015v1 Announce Type: new Abstract: Despite impressive multilingual capabilities, large language models (LLMs) remain poorly evaluated on literary knowledge in non-English languages. We introduce PersLitEval, a benchmark of 4,514 Persian literature multiple-choice questions across eight fine-grained categories spanning spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from materials for the Konkur university entrance examination. We evaluate six LLMs across ten prompting strategies, revealing striking category-level disparities a
The rapid development and widespread adoption of LLMs necessitate a deeper understanding of their capabilities and limitations across diverse linguistic and cultural contexts, particularly as global AI ambitions grow.
This benchmark highlights a critical gap in LLM evaluation beyond English, underscoring the need for culturally and linguistically specific datasets to develop truly robust and equitable AI systems.
The introduction of PersLitEval provides a standardized tool to rigorously assess LLM performance on complex non-English literary knowledge, enabling more targeted development and improvement of multilingual models.
- · Iranian AI developers
- · Multilingual NLP researchers
- · Local language preservation efforts
- · Educational technology sector in non-English speaking countries
- · English-centric LLM development paradigms
- · Organizations relying solely on generic LLM evaluations
It provides a clear evaluation tool for LLMs on non-English literary tasks.
This will likely spur increased investment and research into culturally specific AI models and benchmarks for other languages.
The enhanced capability of LLMs in diverse languages could accelerate the development of localized AI applications and reduce digital divides based on linguistic barriers.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL