Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
arXiv:2606.02255v1 Announce Type: new Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled. We provide the first large-scale, task-level audit of human annotation reporting across major NLP venues, asking which annotation details are documented, which are missing, and how reporting varies across time, topic, venue, and intended use of human judgment. We introduce a unified taxonomy of annotation-reporting practices and
The rapid expansion of AI research and deployment, particularly in NLP, necessitates a more rigorous and transparent approach to data provenance and quality, which this paper aims to address. The publication date in the future suggests proactive research into emerging issues in AI development.
This research highlights a critical vulnerability in AI development: the unseen and often poorly documented human labor underpinning vast AI datasets. Poor annotation practices can lead to biased, unreliable, and uninterpretable AI models, impacting trustworthiness and effectiveness across sectors.
A large-scale audit of NLP annotation practices will establish benchmarks and expose systemic weaknesses in data collection, potentially leading to new industry standards for transparency and quality control in AI dataset creation.
- · AI ethics researchers
- · Dataset quality assurance providers
- · Responsible AI developers
- · Users of NLP applications
- · Developers relying on opaque annotation processes
- · Companies with low data transparency standards
- · Providers of poor-quality annotation services
Increased scrutiny and demand for transparency in AI dataset annotation practices will become standard.
New tools and services will emerge to verify and audit human annotation processes and outcomes, potentially raising data costs for AI development.
Enhanced data transparency could reduce AI biases and improve model reliability, leading to broader trust and adoption of AI systems in sensitive applications like healthcare or legal analysis.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL