ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
arXiv:2510.16928v3 Announce Type: replace Abstract: Existing benchmarks for large language models (LLMs) are largely restricted to high- or mid-resource languages, and often evaluate performance on higher-order tasks in reasoning and generation. However, plenty of evidence points to the fact that LLMs lack basic linguistic competence in the vast majority of the world's 3800+ written languages. We introduce ChiKhaPo, consisting of 8 subtasks of varying difficulty designed to evaluate the lexical comprehension and generation abilities of generative models. ChiKhaPo draws on existing lexicons, mo
The proliferation of LLMs and increasing research into their underlying capabilities and limitations necessitate improved evaluation benchmarks beyond high-resource languages.
This benchmark highlights a critical gap in current LLM capabilities regarding global linguistic diversity, impacting real-world applicability and equitable AI development.
The introduction of ChiKhaPo provides a standardized tool to rigorously evaluate and compare LLMs on multilingual lexical competence, pushing for more inclusive AI.
- · Developers of truly multilingual LLMs
- · Low-resource language communities
- · AI ethics and fairness researchers
- · LLMs with poor multilingual support
- · Benchmarks limited to high-resource languages
Increased focus and investment in improving LLM performance for a wider range of human languages.
Development of new architectural approaches and pre-training strategies specifically optimized for multilingual lexical understanding.
Enhanced global adoption of AI technologies as they become genuinely useful and culturally relevant for diverse populations, potentially reducing digital divides.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL