Nature Medicine study finds general-purpose LLMs outperform specialized clinical AI on medical benchmarks

Medicine discovers the bitter lesson: frontier LLMs (here GPT 5.2, Opus 4.6, Gemini 3.1) outperform specialized "clinical AI" (e.g. OpenEvidence) in a blind test.

Even funnier that hospital IT are more likely to approve the *specialized* versions despite them being worse.

For medical information, general AI frontier models (Google, OpenAI, Anthropic) outperformed specialized @EvidenceOpen and @UpToDate as assessed by 12 US clinicians, randomized and blinded to which model and extensive testing/benchmarks. This was not anticipated. @NatureMedicine https://www.nature.com/articles/s41591-026-04431-5

There has been a push to use OpenEvidence AI for doctors. But this paper suggests general models are much better: “Frontier LLMs outperformed clinical AI tools in all three evaluations. Clinical AI tools performed comparably to auto-enabled Google Search AI Overview on the RCQ.”

>65% of US physicians use OpenEvidence, with 27 million prompts in April https://www.nbcnews.com/tech/tech-news/openevidence-ai-doctor-medical-physician-login-app-what-npi-uptodate-rcna341064

Here is the performance breakdown for each model's blinded assessment for 4 major tasks: (1) clinical correctness, (2) completeness, (3) safety, and (4) clarity.

Why this is a big deal.

I'm not all that surprised by this. Sutton's bitter lesson tells us that generalist models trained on far more data outperform narrower models with less data.

A Nature Medicine study found general-purpose LLMs are now outperforming dedicated medical AI products on physician-reviewed clinical tasks.

The authors compared OpenEvidence and UpToDate Expert AI with GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 on medical exam questions, clinician-style answers, and real questions doctors asked during care.

In 100 de-identified physician questions from live clinical use, blinded clinicians again preferred the frontier models, especially on completeness and clarity,

This was always going to be the case and continues to befuddle that “sophisticated” investors think proprietary data will help.

That’s only true in very specific areas.

But the marketing value of domain specific data still matters to end customers.

"Experts" really do not want to believe this (see Topol's "this was not anticipated", even though this is just Rich Sutton 101), nor do IT departments, but they'll learn eventually I guess

@ramez Actually if you look at the evals at the end, OE wins on accuracy and only loses on writing style (weird eval to include) and follow up prompt.

if you're busy, all you need to look at in my opinion are fig. 2 (c-d)

Yes. Widely relying on this for patient diagnosis and management w/o prospective, rigorous assessment for real world tasks until now. As the authors point out: "scale, alignment and cross-domain reasoning may outweigh domain-specific tuning as determinants of medical competency for particular tasks"

For the past 18 months, I’ve worked on LLMs in Nursing; and I can say that generalist models (GPT-5, Gemini 3pro) always beat health specialist LLMs on my benchmarks and there are two reasons for this:

1) Reinforcement Learning: Because factual reasoning is distilled into general LLMs through on-policy RL with verifiable rewards, they often outperform specialised LLMs that were often usually adapted through LORA supervised fine tuning off-policy.

2) Parameter Size: Most general LLMs have weights to the magnitude of several trillion parameters, and will always make them better that custom-domain one’s with smaller parameters

But I'm not sure how much to trust these results. OE has lots of MDs in-house and I'd be surprised if they are shipping a product that their customers really think is worse. And ultimately this is a pretty non-verifiable domain, where what doctors think is the only real eval.

This exemplifies the paradox of medical AI implementation https://erictopol.substack.com/p/the-paradox-of-medical-ai-implementation

Last thought: is MedGemma actually frontier at anything? It seems like this paper is largely an indictment of specialist harnesses, is the same thing true for specialist LLMs?

https://www.nature.com/articles/s41591-026-04431-5

@nabeelqu This is 12 clinicians, of un-reported specialties & training, rating the responses of an LLM to *someone else’s* question, potentially in a totally different area of medicine. Not sure this tells us much but that frontier models are better at generating widely agreeable answers.