arXiv:2606.28960v1 Announce Type: new Abstract: Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice. We report a blinded evaluation built on 620 Real-world Point-Of-Care Queries (Real-POCQi) submitted to the OpenEvidence (OE) platform by physicians spanning 30 specialties, as well as 187 questions from HealthBench. 149 practicing physicians across 36 states made head-to-head comparisons between answers from three frontier general-purpose models (Claude Opus
As AI tools become ubiquitous in clinical settings, evaluating their real-world performance with actual medical queries is crucial for adoption and safety, differentiating from hypothetical evaluations.
This study provides critical, real-world validation data for AI tools in healthcare, influencing physician trust, regulatory frameworks, and market acceptance for clinical AI applications.
The focus for AI evaluation shifts from theoretical or benchmark questions to practical, point-of-care queries, demanding more robust and context-aware AI models for medical use.
- · AI developers with robust, empirically validated clinical tools
- · Healthcare providers adopting validated AI for improved diagnostics/workflows
- · Patients benefiting from more accurate and reliable AI medical advice
- · AI developers whose tools fail real-world clinical benchmarks
- · Traditional diagnostic methods if AI proves superior and accessible
Physicians gain more trusted AI assistants, potentially improving diagnostic accuracy and efficiency across specialties.
Regulatory bodies might develop new standards for clinical AI certification based on real-world performance metrics, influencing future development cycles.
The widespread adoption of validated clinical AI could lead to a redefinition of medical training, incorporating AI interaction and oversight as core competencies.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI