EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
arXiv:2606.30256v1 Announce Type: new Abstract: Safety benchmarks often buy scalability by fixing the prompt, the language, and the turn structure. For emotional-support chatbots, that bargain hides precisely where safety failures emerge: across a multilingual, multi-turn crisis conversation. We present EMPATH, a benchmark for safety evaluation of emotional-support chatbots. An auditor model role-plays help-seeking users, generating multi-turn conversations from 140 seed instructions and 34 personas. A judge model scores each full transcript against 19 metrics across five dimensions: crisis ha
The proliferation of advanced AI models highlights the urgent need for robust and scalable safety evaluation, especially for sensitive applications like emotional support.
This benchmark addresses a critical gap in AI safety, moving beyond simplistic evaluations to assess sophisticated, multi-turn, multilingual interactions crucial for real-world chatbot deployment.
The introduction of EMPATH provides a more comprehensive and realistic method for evaluating the safety of emotional-support chatbots, emphasizing multi-turn and multilingual scenarios.
- · AI safety researchers
- · Emotional support chatbot developers
- · AI ethics organizations
- · Developers relying solely on fixed-prompt safety benchmarks
- · Chatbot providers with inadequate safety protocols
Emotional support chatbots will undergo more rigorous and realistic safety testing, leading to improved reliability and reduced harm.
Increased scrutiny on AI safety in emotional support could accelerate the development of more sophisticated safety alignment techniques and regulatory frameworks.
Higher safety standards for sensitive AI applications may build greater public trust in AI, paving the way for broader adoption in critical domains.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI