arXiv:2607.08256v1 Announce Type: new Abstract: Best-of-$N$ (BoN) inference improves content consistency in zero-shot text-to-speech by selecting from $N$ candidates with an automatic speech recognition (ASR) verifier. We identify an underexplored evaluation confound: a verifier's apparent quality depends strongly on which ASR family judges it. On LibriSpeech-PC test-clean~\citep{librispeechpc} with F5-TTS~\citep{f5tts}, verifier rankings reverse across Whisper, wav2vec~2.0, and HuBERT evaluators, and same-family verifier-evaluator pairs recover 2-3$\times$ more oracle headroom than cross-fami
The paper identifies a crucial methodological flaw in current text-to-speech (TTS) evaluation just as generative AI models are rapidly advancing and becoming more prevalent.
This finding significantly impacts the reliability of TTS model evaluations, potentially slowing down accurate progress and resource allocation in a critical area of AI development.
The perceived quality and comparative performance of various TTS models, especially those using Best-of-N inference, will likely be re-evaluated based on this observed ASR family alignment confound.
- · ASR model developers supporting diverse 'families'
- · Researchers focused on robust, family-agnostic TTS evaluation
- · Companies investing in multimodal AI verification
- · Open-source TTS and ASR communities
- · TTS models highly tuned to specific ASR evaluators
- · Benchmarks relying on single ASR family evaluations
- · Companies making investment decisions solely on current BoN metrics
More rigorous and diversified ASR-based evaluation methodologies will emerge for TTS.
This could lead to a reprioritization of research directions in TTS, focusing more on cross-evaluator robustness.
The development of 'universal' ASR models or meta-evaluators might accelerate to provide more objective benchmarks across AI modalities.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL