Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science
arXiv:2606.12426v1 Announce Type: cross Abstract: LLM annotators are increasingly used in computational social science (CSS), but it is unclear whether their alignment-shaped errors preserve the empirical conclusions a researcher would report. We audit three open-source 7B instruction-tuned models (Zephyr, Mistral-Instruct, Qwen2.5-Instruct) across six TweetEval tasks under four prompt conditions (72 cells) and find that social-desirability failures do not run in a single direction. Zephyr exhibits leniency bias, systematically under-applying harmful labels (offensive language: false benign ra
The increasing reliance on LLM annotators in computational social science necessitates rigorous auditing to ensure the integrity of research findings, especially as these models become more sophisticated and widely adopted.
Strategic readers should care about the accuracy and bias of LLM annotators because these models are influencing research, policy, and product development, particularly in areas like content moderation and social impact analysis.
This research highlights that not all LLM biases manifest uniformly, indicating a more complex landscape for mitigating social-desirability bias than previously assumed and requiring tailored auditing approaches.
- · AI researchers focusing on bias detection
- · Open-source LLM developers improving alignment
- · Computational social scientists understanding LLM limitations
- · Organizations relying on unverified LLM annotation data
- · Researchers unaware of different bias types in LLMs
Increased scrutiny and demand for robust bias evaluation methods for LLM-powered systems.
Development of new benchmarking datasets and AI models specifically designed to counteract social-desirability bias in annotation tasks.
Potential for regulatory frameworks to mandate bias audits for AI systems used in critical social science applications and public policy formation.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL