Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMs
arXiv:2607.00588v1 Announce Type: new Abstract: Continuous diffusion language models such as ELF report record-low generative perplexity (Gen-PPL). We find a catch: these models repeat far more than human text, and Gen-PPL rewards rather than penalizes that repetition, so its low scores overstate quality. Strip the repetition and ELF-B's Gen-PPL rises from $19.5$ to $27.7$; the smallest model even posts the best Gen-PPL because it repeats most. We trace the repetition to its source: a contractive attractor along a \emph{single direction} in the self-conditioning feedback loop, the loop that fe
This research is emerging as organizations increasingly rely on advanced large language models, highlighting a critical flaw in current evaluation metrics that overstate their true generative quality.
A strategic reader should care because reliance on misleading perplexity scores can lead to overestimates of AI capabilities and misallocation of resources in model development and deployment.
The understanding of leading-edge continuous diffusion LMs changes, revealing that their low perplexity metrics are partly an artifact of repetition rather than genuine fluency or creativity.
- · Researchers developing new evaluation metrics
- · Model developers focused on genuine generative diversity
- · Enterprises seeking more rigorous AI quality assessment
- · Models that rely on repetition for low perplexity scores
- · Organizations basing critical decisions on current Gen-PPL metrics
Leading AI models will face scrutiny regarding their actual generative quality beyond conventional perplexity metrics.
There will be increased investment in developing and adopting richer, more nuanced evaluation frameworks for LLMs that penalize repetition and reward diversity.
This could lead to a 'perplexity reset' where current state-of-the-art models are re-evaluated, potentially shifting the competitive landscape and influencing future AI research directions.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL