arXiv:2606.14278v1 Announce Type: new Abstract: Large language models (LLMs) are now widely used as automatic judges for open-ended instruction-following evaluation. This practice is convenient, scalable, and often more semantically aware than reference-based metrics, but it also introduces a new reliability question: does a judge evaluate the quality of an answer, or does it also react to the language in which the comparison is presented? We propose Judge-LS, a lightweight meta-evaluation protocol that transforms LLMBar response-pair items into English, Chinese, and Chinese-English language-s
The proliferation of LLMs as evaluators necessitates rigorous scrutiny into their biases, especially as multilingual applications become more common.
The reliability and impartiality of LLM-based judgments are critical for fair and consistent evaluation of AI systems, impacting development cycles and competitive analysis.
This research introduces a standardized meta-evaluation protocol to uncover language-based biases in LLM judges, prompting developers to account for these subtle influences.
- · AI ethicists
- · Multilingual AI developers
- · LLM evaluation platforms
- · Academic researchers
- · Developers of biased LLM judges
- · Uncritical adopters of LLM-as-a-Judge
LLM evaluations will increasingly incorporate language-invariance testing as a standard practice.
Improved understanding of linguistic bias will lead to the development of more robust and culturally neutral LLM benchmarks.
The pursuit of language-agnostic AI evaluation could foster a more equitable global AI development landscape.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL