arXiv:2605.20642v1 Announce Type: new Abstract: When annotators disagree, that disagreement can reflect epistemic uncertainty rather than simple label noise. We study hard-label delivery as an alternative to the usual choices of collapsing votes to a single label or training directly on the empirical soft-label distribution. We focus on two primary hard-label methods: multipass, which cycles through observed votes while keeping the dataset size fixed, and stochastic label sampling (SLS), which samples one label per example at the start of each epoch. On CIFAR-10H, we find that when only a smal
This research addresses fundamental challenges in machine learning training with uncertain human-annotated data, a pervasive issue as AI systems increasingly rely on diverse and often conflicting human input.
Improved methods for handling annotator disagreement can lead to more robust and accurate AI models, reducing training costs and increasing reliability in real-world applications where data quality is paramount.
The understanding of how to effectively use 'soft labels' and manage epistemic uncertainty in training data is evolving, potentially leading to more sophisticated data labeling and model training strategies.
- · AI researchers
- · Data annotation services
- · Industries relying on human-annotated datasets
- · AI models reliant on simplified label aggregation
More sophisticated approaches to data labeling and model training become standard in academic and industrial AI.
AI systems deployed in real-world scenarios exhibit greater resilience to noisy or subjective human input.
The development of AI agents that can learn effectively from nuanced and conflicting human feedback accelerates, extending their capabilities.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG