Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
arXiv:2605.28301v1 Announce Type: new Abstract: Chain-of-thought (CoT) distillation trains a smaller model to imitate a teacher's reasoning trace, but it is typically evaluated by final-answer metrics including accuracy. We ask whether gains in answer quality are accompanied by improvements in the trace. In medical QA, where short answer options can leave a richer clinical justification under-specified, a Qwen3-8B student distilled from a DeepSeek-V3-family teacher improves on MedQA-USMLE answer metrics (SC@64 74.7% to 84.4%; expected calibration error (ECE) 0.096 to 0.034). Yet under a Kimi-K
This research provides a timely audit of current AI distillation techniques, as the industry increasingly focuses on deploying smaller, more efficient models.
It highlights a critical discrepancy where improved final answer accuracy in distilled models does not necessarily equate to improved underlying reasoning, posing challenges for high-stakes applications like medicine.
The focus might shift from purely answer-based metrics to more comprehensive evaluations that scrutinize the step-level reasoning of AI models, especially in critical domains.
- · AI evaluation companies
- · Developers of reasoning-focused AI architectures
- · DeepSeek-family models
- · Models relying solely on distillation for reasoning
- · Users prioritizing accuracy over explainability
- · Qwen3-8B in medical QA
AI developers will need to refine distillation methods to ensure reasoning quality scales with accuracy.
Increased scrutiny on the black-box nature of AI reasoning will drive demand for more interpretable and robust AI systems.
Regulatory bodies might introduce new standards for AI deployment, especially in fields like medicine, mandating transparent reasoning traces.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI