arXiv:2606.24648v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures
The proliferation of LALMs for speech generation and evaluation creates an immediate need for robust, unbiased assessment benchmarks that extend beyond basic naturalness.
A strategic reader should care because accurate paralinguistic evaluation is crucial for the development of sophisticated, human-like AI speech applications, impacting user experience and the effective deployment of AI agents.
This benchmark shifts the focus from holistic speech quality to fine-grained paralinguistic dimensions, highlighting current LALM limitations and guiding future research into more nuanced audio understanding.
- · AI researchers focusing on speech understanding
- · Companies developing advanced voice AI
- · Developers of more sophisticated LALM evaluation metrics
- · LALM developers over-relying on current, holistic evaluation methods
- · Applications requiring precise paralinguistic control without robust evaluation
The benchmark reveals significant gaps in LALM judges' ability to evaluate paralinguistic distinctions, necessitating improved model architectures and training data.
Enhanced paralinguistic evaluation capabilities will lead to more expressive and contextually appropriate AI-generated speech, blurring the line between human and artificial voices.
The ability to accurately model and generate fine-grained paralinguistic cues could enable highly personalized and emotionally intelligent AI agents, transforming human-computer interaction and potentially raising ethical considerations.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL