arXiv:2606.00285v1 Announce Type: new Abstract: Large-scale multilingual bitext often contains two distinct problems: non-parallel sentence pairs and low-quality translations. We decompose model-based assessment for such data into two independent components: parallelism assessment with multilingual embeddings and reference-free quality estimation (QE). For parallelism, we benchmark four embedding models on FLORES-200 and BOUQuET retrieval tasks, covering 6,654 source--target directions in our target language-pair inventory. For QE, we evaluate nine reference-free evaluators on professional FLO
The proliferation of multilingual AI models necessitates robust methods for quality assurance of the vast parallel data required for their training and fine-tuning.
Improving the quality assessment of massively multilingual parallel data directly impacts the performance and reliability of large-scale AI models, particularly in diverse language environments.
This research provides a more systematic and decompositional approach to evaluating multilingual bitext, potentially leading to more accurate and efficient data curation for AI development.
- · AI developers
- · Multilingual AI platforms
- · Language technology companies
- · Researchers in NLP
- · Providers of low-quality parallel data
- · AI models trained on unvetted data
More reliable and performant multilingual AI models will emerge due to better training data quality.
The cost and time required for curating high-quality multilingual datasets may decrease, accelerating AI development in new language pairs.
Improved multilingual AI capabilities could enable broader adoption of AI in diverse linguistic markets and reduce biases stemming from poor data quality.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL