arXiv:2605.27239v1 Announce Type: new Abstract: Annotation quality is difficult to sustain when campaigns span weeks or months with small annotator pools. We present a Setswana sentiment dataset of 3,565 tweets annotated by three native-speaker annotators across eight batches and examine why inter-annotator agreement (IAA) declines over time. Despite an aggregate Randolph's free-marginal Kappa of $\kappa = 0.76$, "excellent," per-batch $\kappa$ falls by more than 32 points across the annotation task. Through six targeted analyses, we find that (i) label confusion concentrates on the negative/n
The proliferation of AI models, especially large language models, makes robust and consistent data annotation critical for model performance and reliability, yet annotation quality remains a persistent challenge.
This research provides empirical evidence and specific insights into factors causing annotation decay, offering actionable intelligence for improving dataset quality which directly impacts AI development and deployment.
Understanding of annotation quality degradation is refined to include temporal factors and specific label confusion, enabling more effective strategies for dataset creation and maintenance.
- · AI developers
- · Data annotation platforms
- · NLP researchers
- · Ethical AI advocates
- · AI projects with poorly managed annotation pipelines
- · Models trained on low-quality data
- · Annotation services without quality control
Improved annotation strategies lead to higher quality sentiment datasets.
Better datasets result in more accurate and robust sentiment analysis models, especially for underrepresented languages.
Enhanced AI model performance across various applications due to more reliable training data, fostering greater trust in AI systems.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL