Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization
arXiv:2606.12854v1 Announce Type: new Abstract: Large Language Models such as GPT-4o and GPT-5 achieve strong zero-shot performance on biomedical claim verification, but cost and opacity limit scalable use. We fine-tune three small LLMs: Phi-3-mini (3.8B), Qwen2.5-3B, and Mistral-7B, via QLoRA on SciFact and HealthVer, providing the first study of QLoRA models against GPT-4o and fine-tuned BioLinkBERT encoders. Mistral-7B QLoRA surpasses both GPT-4o and GPT-5 (up to 12% F1 gain) at a fractional cost using just 1,008 training examples. We conduct extensive in-domain and cross-domain evaluation:
Ongoing advancements in efficient fine-tuning techniques for LLMs and increasing computational costs for larger models make this research timely.
This research demonstrates that smaller, fine-tuned LLMs can outperform much larger, more expensive models like GPT-4o and GPT-5 for specific tasks, significantly reducing cost and increasing accessibility.
The paradigm shifts towards fine-tuned smaller models becoming viable, and in some cases superior, to massive foundational models for specialized applications, especially in cost-sensitive and transparency-demanding sectors like biomedical research.
- · Biomedical research
- · Open-source AI models
- · Organizations with limited compute budgets
- · Developers leveraging specialized AI
- · Mega LLMs offering generalized services
- · Cloud providers reliant on high inference costs
- · Companies exclusively investing in large proprietary models
Increased adoption of smaller, specialized LLMs in various domain-specific applications due to cost-effectiveness and performance.
A potential reduction in the dominance of a few large AI labs as niche models gain competitive advantages.
Democratization of advanced AI capabilities, potentially leading to a wider range of impactful applications beyond current commercial offerings.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL