Who Heals Better: General AI or a Narrow Specialist express
Every week, doctors ask AI assistants millions of questions at the bedside. But these tools are usually tested on exam-style problems, not on what clinicians actually ask in practice. A team of researchers set out to fix that.
They collected 620 real questions from physicians across 30 specialties — from cardiology to oncology. These questions were submitted to the OpenEvidence platform. Then, 149 practicing doctors from 36 states compared answers from three flagship general models (Claude Opus 4.8, Gemini 3.1 Pro, GPT-5.5) and a specialized medical tool (OE). Each judge was from the same specialty as the physician who asked the question.
They evaluated along five axes: accuracy, clinical usefulness, source quality, verifiability, and completeness. The specialized tool won on every point. The difference between wins and losses ranged from 25 to 39 percentage points — with overwhelming statistical significance (p<0.001). The result held regardless of whether citations were shown, how long the answers were, or whether the judge had used the tool before.
There are two takeaways. First, medical AI should be tested on real questions with input from doctors in the same specialty — otherwise the picture is incomplete. Second, the specialized tool's advantage doesn't mean general models are useless. It shows that targeted engineering yields tangible benefits for users. Like a well-tuned stethoscope picks up what a general-purpose microphone misses.
Original abstract
as written by the authors arXiv:2606.28960
Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice. We report a blinded evaluation built on 620 Real-world Point-Of-Care Queries (Real-POCQi) submitted to the OpenEvidence (OE) platform by physicians spanning 30 specialties, as well as 187 questions from HealthBench. 149 practicing physicians across 36 states made head-to-head comparisons between answers from three frontier general-purpose models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5) and a specialized clinical tool (OE), with graders matched to each question's specialty. When comparing answers along five dimensions relevant to clinical decision support -- accuracy, clinical utility, source quality, verifiability, & completeness -- physicians scored the specialized tool highest on all axes; in the primary analysis on Real-POCQi, win differences (margins between win and loss rates) ranged from 25 to 39 percentage points (p<0.001). Results remained consistent in sensitivity analyses stratifying by citation display, answer length, OE-user status, and Real-POCQi versus HealthBench. In parallel, LLM judges were found to systematically differ from expert judges, though both generally agreed on the best model. These findings underscore two conclusions: (i) AI tool evaluations should reflect real-world query distributions and use expert judges that mirror the specialization defining modern medicine and (ii) the consistent advantage of the specialized tool over general-purpose models does not necessarily mean that the latter cannot serve similar purposes, but that targeted engineering and customization can yield meaningful gains in performance for its users. We release Real-POCQi as a public benchmark, as well as the prespecified statistical analysis for reproducing results of this study.
The margin in wins reached 39 percentage points: that's like one team winning 7 out of 10 matches while the other wins only 3.