LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context
arXiv:2606.24585v1 Announce Type: new Abstract: While the validity of LLMs' use in the legal context remains subject to ethical and legal debate, legal professionals are already experimenting with personal LLMs, if only for translation and reformulation. However, even such a seemingly innocuous use can introduce biases through case processing speed if LLM assistants selectively refuse assistance on certain topics. To better anticipate such biases, we investigate several modern small LLMs that are most likely to be used as on-device assistants, to assess the impact of overrefusal on legal promp
The proliferation of accessible, smaller LLMs on premises makes their direct application in professional, ethics-sensitive fields like law a pressing concern, requiring immediate investigation into their biases.
This research highlights a critical bias in LLMs, 'overrefusal,' which can disproportionately impact access to legal assistance and introduce subtle but significant distortions in justice systems.
The findings change the understanding of how LLMs, even when used for seemingly innocuous tasks, can introduce new forms of bias and necessitate re-evaluation of deployment strategies in sensitive sectors.
- · AI ethics researchers
- · Legal tech developers focusing on explainability
- · Legal professionals relying on un-audited LLMs
- · AI companies neglecting bias in small models
Small, on-premise LLMs trained on legal data show a tendency to 'overrefuse' assistance when prompted with legal context.
This overrefusal leads to unequal access to automated legal assistance, potentially exacerbating biases within the legal system based on case type or user query sophistication.
Regulatory bodies may implement new guidelines or certifications specifically for LLMs used in legal contexts, focusing on refusal rates and bias audits for deployment.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI