arXiv:2607.05031v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to produce test oracles, the part of a test that decides whether observed behavior is correct. Yet a clear account of where these oracles draw their authority is missing. Prior secondary studies organize the area by oracle form or by LLM technique. None organizes it by the source of the verdict's authority, the property that governs how far a verdict can be trusted. This article presents a systematic literature review, conducted and reported under the PRISMA 2020 guidelines. From 2,436 records,
The proliferation of LLMs in software development, particularly for testing, necessitates a deeper understanding of their reliability and trustworthiness, leading to this systematic review.
As LLMs increasingly automate critical software functions, understanding the 'source-of-authority' for their test oracles is crucial for ensuring software quality, safety, and accountability.
This research introduces a new framework for evaluating LLM-based test oracles based on their source of authority, potentially redefining how software testing and quality assurance in AI are approached.
- · Software QA professionals
- · AI safety researchers
- · Developers of robust LLM-based testing tools
- · Developers of unreliable LLM testing solutions
- · Organizations with inadequate AI testing protocols
Improved methodologies for validating the correctness of software and systems developed with or through AI.
Increased trust and adoption of AI-driven development tools as their reliability can be more systematically assessed.
The development of industry standards and regulations specifically for AI-generated code and test outputs, emphasizing verifiable authority.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI