Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
arXiv:2606.30790v1 Announce Type: new Abstract: Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities. While Large Language Models (LLMs) perform strongly on monolingual and native-script benchmarks, their ability to follow instructions and reason over RCM-based content remains largely unexplored. To this end, we introduce the Indi-RomCoM benchmark for facilitating systematic evaluation on Indic Romanized Code-Mixed instructions. Our benchmark spans seve
The proliferation of LLMs and their growing adoption in diverse linguistic contexts necessitates robust evaluation benchmarks to ensure their reliability and fairness.
This benchmark highlights a critical gap in LLM performance for Romanized Code-Mixed languages, which represent a significant portion of global communication, particularly in India.
LLM developers now have a standardized tool to systematically assess and improve their models' understanding and generation capabilities for Romanized Indic-English content.
- · Indic language speakers
- · Multilingual LLM developers
- · AI researchers in NLP
- · Companies targeting Indian markets
- · LLMs with poor multilingual understanding
- · Monolingual AI content strategies
Improved performance of LLMs in Romanized Indic-English contexts, leading to better user experiences.
Increased investment and research into multilingual AI models, particularly for low-resource and code-mixed languages.
Potential for new AI applications and services tailored to multilingual populations, fostering greater digital inclusion and economic opportunities in these regions.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL