Koshur Diacritizer: A Byte-Level Sequence-to-Sequence Model for Kashmiri Diacritic Restoration
arXiv:2606.15883v1 Announce Type: new Abstract: Kashmiri, an Indo-Aryan language written in a modified Perso-Arabic script, frequently omits diacritic marks in digital text, creating ambiguity and challenging downstream NLP applications. We present Koshur Diacritizer, a ByT5-small byte-level sequence-to-sequence model for restoring diacritics in Kashmiri text. To support this task, we release a publicly available dataset of 23.7k aligned undiacritized diacritized Kashmiri sentence pairs. The proposed framework combines script-aware normalization, alignment validation, and skeleton-preserving i
The proliferation of digital text and the increasing demand for NLP applications necessitates robust tools for under-resourced languages like Kashmiri, aligning with broader efforts in linguistic AI development.
This work directly addresses a critical gap in language technology for Kashmiri, enabling better accessibility and usability of the language in digital formats and supporting its long-term digital preservation and utility.
The availability of Koshur Diacritizer and a new dataset significantly lowers the barrier for developing advanced NLP tools for Kashmiri, potentially expanding its digital footprint and integration into AI applications.
- · Kashmiri language users
- · NLP researchers
- · Local cultural organizations
Improved NLP accuracy and accessibility for Kashmiri language text.
Increased digital content creation and consumption in Kashmiri due to reduced ambiguity.
Potential for sovereign AI initiatives by other less-resourced language communities, driving localized AI development.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL