Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
arXiv:2601.22588v2 Announce Type: replace Abstract: Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesi
The increasing cost and sensitivity of large language models for evaluation necessitates exploring more efficient alternatives, driving current research into their foundational mechanisms.
This research suggests a more cost-effective and robust method for evaluating AI models, potentially democratizing access to high-quality AI development and reducing reliance on expensive, opaque LLM judges.
The paradigm for AI model evaluation could shift from relying on large, generative LLMs to more efficient small language models leveraging internal representations, offering greater transparency and resource optimization.
- · AI developers (SMBs)
- · AI research institutions
- · Cloud infrastructure providers (optimised compute)
- · Large language model providers (for judging tasks)
- · Legacy AI evaluation methodologies
Small language models could become the preferred standard for AI evaluation due to their efficiency and transparency.
Reduced computational costs for AI development and iteration could accelerate innovation across various applications.
This could lead to a proliferation of specialized, highly efficient AI models due to lower barriers to effective evaluation and refinement.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL