Who Heals Better: General AI or a Narrow Specialist express

‹›

Every week, doctors ask AI assistants millions of questions at the bedside. But these tools are usually tested on exam-style problems, not on what clinicians actually ask in practice. A team of researchers set out to fix that.

They collected 620 real questions from physicians across 30 specialties — from cardiology to oncology. These questions were submitted to the OpenEvidence platform. Then, 149 practicing doctors from 36 states compared answers from three flagship general models (Claude Opus 4.8, Gemini 3.1 Pro, GPT-5.5) and a specialized medical tool (OE). Each judge was from the same specialty as the physician who asked the question.

They evaluated along five axes: accuracy, clinical usefulness, source quality, verifiability, and completeness. The specialized tool won on every point. The difference between wins and losses ranged from 25 to 39 percentage points — with overwhelming statistical significance (p<0.001). The result held regardless of whether citations were shown, how long the answers were, or whether the judge had used the tool before.

There are two takeaways. First, medical AI should be tested on real questions with input from doctors in the same specialty — otherwise the picture is incomplete. Second, the specialized tool's advantage doesn't mean general models are useless. It shows that targeted engineering yields tangible benefits for users. Like a well-tuned stethoscope picks up what a general-purpose microphone misses.