A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays
arXiv:2605.25652v1 Announce Type: new Abstract: Free-form legal essay evaluation in NLP treats expert inter-rater stability as a single ceiling number, and treats LLM-judge agreement with that ceiling as evidence of judge stability. We test both assumptions on the Thai bar examination through an identical-inputs protocol: three Bar Council-trained examiners (A, B, C) and a 26-LLM judge panel score the same 15 cross-graded answers from the same four inputs (question, official Bar Council grading regulation, gold answer, candidate answer). The headline finding is asymmetric. On 10 of 15 cells wh
The proliferation of LLMs creates an immediate need to understand their reliability and stability in critical, expert-driven tasks like legal evaluation, especially as enterprises explore their integration into sensitive workflows.
This study provides crucial empirical data on LLM judge performance against human experts in a high-stakes, free-form text evaluation, informing the realistic expectations and limitations for AI integration in professional domains.
The understanding that LLM judge 'stability' might not directly equate to human expert inter-rater agreement, and that their evaluation discrepancies are asymmetric, changes how AI performance in subjective tasks should be assessed.
- · AI ethics and safety researchers
- · Legal tech developers focusing on human-in-the-loop systems
- · Institutions developing human oversight protocols for AI
- · Platforms overpromising LLM autonomy in complex legal analysis
- · Organizations relying solely on LLM judges for high-stakes evaluations
- · Professionals unfamiliar with AI limitations
The findings will temper expectations for fully autonomous LLM judging in nuanced legal contexts and similar expert domains.
This could lead to increased focus on hybrid human-AI evaluation systems where LLMs act as assistants rather than sole decision-makers.
These insights may inform regulatory frameworks and accreditation processes for AI tools used in professional fields, emphasizing stability and agreement against human benchmarks rather than just 'accuracy'.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL