Whose Name Comes Up? II: Benchmarking and Intervention-Based Auditing of LLM-Based Scholar Recommendation
arXiv:2602.08873v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are now used for academic expert recommendation. Existing audits typically evaluate such recommendations in isolation, ignoring end-user inference-time interventions. Thus, it remains unclear whether failures (e.g., refusals, hallucinations, uneven coverage) stem from model choice or deployment decisions. We introduce LLMScholarBench, a benchmark for auditing LLM-based scholar recommendation that jointly evaluates model infrastructure and end-user interventions across multiple tasks. LLMScholarBench measures
The increasing deployment of LLMs across various critical applications, including academic recommendation, necessitates robust evaluation and auditing frameworks to address emerging biases and failures.
This development highlights the ongoing challenge of ensuring fairness and accuracy in AI systems, especially as LLMs are integrated into systems that influence professional reputation and resource allocation.
The introduction of a specific benchmark allows for systematic identification and mitigation of biases in LLM-based scholar recommendation, moving beyond isolated evaluations to address real-world deployment challenges.
- · AI ethicists
- · Fairness in AI researchers
- · Academic institutions (indirectly)
- · Developers of biased LLM applications
- · Academic recommenders (if relying solely on un-audited LLMs)
Increased scrutiny and demand for transparent AI auditing tools for LLM-based systems.
Development of industry standards and regulatory guidelines for ethical AI deployment in sensitive areas like professional recommendation.
A shift in how academic contributions and expertise are recognized, with potential for revised metrics that account for AI-driven amplification or suppression.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI