Does Topic Sentiment Cause Perceived Ideology? Comparing Human and LLM Annotations in Political News Articles
arXiv:2606.06715v1 Announce Type: cross Abstract: We ask whether topic sentiment has a causal effect on perceived political ideology, and whether the answer depends on who assigns the ideology label. Using articles from AllSides, paired with shared sentiment annotations from Llama-3.3-70b-versatile, we compare ideology labels from expert human annotators, GPT-4o-mini (baseline and finetuned), and Llama-3.3-70B. We apply Double Machine Learning (DML) and community-level mediation analysis across all four annotation paradigms. Human annotations yield no significant causal effects at the communit
The proliferation of advanced LLMs like Llama-3.3-70b-versatile and GPT-4o-mini makes their application to complex tasks like political ideology assessment a current research frontier. The study leverages existing political news datasets to compare human and AI annotations.
This research explores fundamental differences in how humans and advanced AI perceive and attribute political ideology, highlighting potential biases and limitations in AI applications for sensitive social and political analysis. The findings could influence the design and deployment of AI systems in areas like content moderation, political campaigning, and news analysis.
Our understanding of AI's capability to replicate human nuanced understanding of political sentiment and ideology is updated, revealing that AI may not interpret causality or attribute ideology in the same way as humans. This suggests caution in deploying LLMs for tasks requiring nuanced social judgment without human oversight or specialized fine-tuning.
- · AI ethics researchers
- · AI model developers focused on bias mitigation
- · News organizations seeking robust annotation methods
- · Platforms relying solely on generic LLM outputs for nuanced political analysis
- · Organizations using off-the-shelf LLMs for content moderation tasks without furt
The study reveals divergences between human and LLM perceptions of political ideology, particularly regarding the causal effect of topic sentiment.
This could lead to increased scrutiny and demand for explainability and bias mitigation in AI models used for political or social analysis.
Future AI development for sensitive applications may shift towards creating models with more sophisticated, human-aligned causal reasoning, or a hybrid human-AI annotation approach could become standard.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG