The medical models didn't win

I lead DevOps in healthcare IT, and I keep hearing the same assumption: for clinical work, you want a medical model. A model fine-tuned on medical text, the reasoning goes, must read a patient’s chart better than a general-purpose one.

So I tested it — on my own hardware and in the cloud, with synthetic data, on one narrow job: answering questions about a long patient chart.

Why open models, and why these

A startup can reasonably call the best hosted API and move on. An enterprise healthcare IT provider usually weighs two routes: run models in-house, where the data already lives, or partner with a large cloud vendor (Google, Microsoft, Amazon) under the agreements healthcare data requires. Open models are what make the first route possible: you can download them, run them on your own hardware, and keep the exact version you validated. So that’s what I tested, with frontier cloud models as a ceiling. Which route makes sense, and at what scale, is a later post in this series: Build or rent?

Four of the eight models are Chinese-origin: two Qwen models from Alibaba, and two medical models built on Qwen, HuatuoGPT-3 and Baichuan-M2. They’re here because the fair test of medical training is a medical model against the exact base it was built from, and many open medical models are built on Qwen. Including them isn’t a recommendation to deploy them. Country of origin is a legitimate procurement check, and it gets its own place in the series’ last post: Choosing a model is a product decision.

The test

I generated 12 synthetic patients with Synthea — no real records anywhere — as plain-text charts ranging from 2,500 to 19,000 tokens. Then I wrote 163 questions whose answers I could compute directly from the data: current medications, latest lab values, lab trends, visit counts, and a category I’ll come back to — unanswerable questions about tests that aren’t in the chart at all.

Eight open models, in ten configurations, same prompt, scored automatically against the computed answers. I ran the full set locally, on two used data-center GPUs, then re-ran several of the models hosted through OpenRouter — to be sure the numbers weren’t an artifact of my setup. They landed within two or three points of local, and a frontier model I spot-checked agreed.

The ranking

A general-purpose model tied for the top. Qwen3-27B and HuatuoGPT-3 — a medical fine-tune of Qwen — both hit 93%. The medical models landed all over the board: MedGemma-27B at 69%, MedGemma-4B at 46%.

The cleaner way to read this is to compare each medical model against the exact base model it was built on. Done that way, the medical training was worth 0 to 4 percentage points on this task. Meanwhile, turning on “thinking” mode — letting the model reason before answering — was worth 9. A setting I could turn on — no new model, no retraining, no license — beat the fine-tuning everyone assumes you need. It isn’t free, though. Thinking roughly tripled the compute each answer took (64–81 minutes for the test set on my box, versus about 20) and burned about five times the energy. That’s not money spent on a model; it’s capacity — fewer answers per GPU per hour — which is exactly the tradeoff the later posts on cost and on serving at scale have to price in.

Where the medical tuning does — and doesn’t — pay off

That doesn’t make medical models bad. It means they’re best known for other jobs — and when I tested those, the picture flipped.

On medical images, the fine-tuning did pay off — but the win is narrower than “wins at images” makes it sound, and the narrowness is the real story. The small model, MedGemma-4B, beat its base by 21 to 22 percentage points on the standard radiology question sets. But on a harder CT/MRI set the gain shrank to about 7 points — and all of it came from a single skill: naming which organs are present. On the skills that carry real clinical weight — locating a structure, reading what’s inside a marked region, judging relative position — the medical tuning bought nothing. And at the larger 27B size, the medical model scored below the general one it was built from. So “wins at images” really means a small model, on the easiest kind of image question, on the standard tests — not a broad imaging advantage.

On medical exam questions, with both models given the same chance to reason, it was a tie. (My first runs showed big medical exam wins; those turned out to be artifacts — one model thinking while the other answered blind, or running out of room to finish. Fixing the test erased the gap.)

So the honest headline isn’t “medical models are worse” — and it isn’t “medical models win at images” either. It’s narrower, and less convenient for both camps: the fine-tuning gives a small model a bump on the simplest image task, does little for the harder ones, can reverse at larger sizes, and barely moves the needle on reading text in a chart.

The failure worth naming

Back to that last question category. Twenty-four of my 163 questions were unanswerable by design: they asked about a test that simply isn’t in the chart. The only correct response is to say so — “not in the record.” Every model 27B and up got all 24 right, with zero invented values.

That sounds unremarkable until you notice what the wrong answer would have been: a confident, plausible lab value that was never drawn. In a patient chart, that’s the scariest failure mode there is, because a fabricated value is indistinguishable from a real one.

It’s also worth being precise about the right answer. “Not in the record” is a claim about the documentation — absence of evidence. It is not “the patient never had this test,” which is a claim about the patient — evidence of absence. A chart can be incomplete; the honest system reports the gap, it doesn’t fill it.

How I’d lead this

If I were steering this for a product team, here’s what I’d want people to take from it:

1. Start with the job, not the model. “Which model is best?” has no answer. “Which model is best at this task, on data shaped like ours?” does.
2. Test before you decide. A few hundred questions with computed answers is cheap next to choosing on reputation.
3. Try the cheap knobs first. Thinking mode cost nothing to acquire — no new model, no training — and still beat the medical fine-tuning on this task. But cheap to adopt isn’t free to run: it roughly tripled the compute per answer and the energy with it, a real capacity cost at scale.
4. Bigger isn’t automatically better. A 2024 70-billion-parameter model lost to a 2026 27B, at a fifth of the speed.
5. Accuracy is one pillar. Cost, where the data goes, the license terms, the supply chain behind the weights, and who’s liable are the rest of the decision. A leaderboard isn’t a model choice.

The caveats

This was lookup and counting, not diagnosis — a deliberately narrow job, chosen because I could grade it exactly. And it isn’t an artifact of aging hardware: I re-ran a model on a newer GPU and 157 of 163 answers matched, moving the score by a single point.

Point 5 above is also a promise about the rest of this series. Performance is the first question, not the only one — and even within performance, this was one task of several; a later post looks at catching errors in a chart, not just reading values back. Cost, data governance, licensing, supply chain and liability each get their own post.

How I ran this

For anyone who wants the method instead of the headline — the part I’d expect a technical reviewer to push on.

The data. Twelve synthetic patients generated with Synthea, rendered as plain-text charts of roughly 2,500 to 19,000 tokens. Nothing real — no PHI, no de-identified records, no scraped notes. Because the charts were generated, I knew every fact in them, which is what made exact automatic grading possible.

The questions. 163 of them, in seven categories, each probing a different lookup skill: current medications, latest lab value, lab trend, diagnosis date, visit count, stopped medications, and “absent” — a test that isn’t in the chart. Every answer was computed directly from the Synthea data, so grading was a numeric or string comparison against ground truth, not a human judgment and not another model’s opinion. Twenty-four of the 163 were “absent,” where the only correct response is to decline.

The models. Eight open models in ten configurations, all run locally under llama.cpp on GGUF weights, the same prompt for every one:

The three Qwen rows are the same weights at different quantization (Q8 vs Q4) and with thinking on or off — that’s how I isolated what each knob was worth rather than guessing at it.

Thinking, and what it actually costs. “Thinking” means letting the model emit reasoning tokens before its answer. It was worth about 9 points on lookup — but it is not free to run. On my box the full set took 64–81 minutes with thinking against about 20 without, and a measurement at the wall plug put thinking at 4.5–5.5× the energy per thousand answers. On fixed hardware that is simply fewer answers per hour: a capacity and cost decision, not a free lunch. It also needs room to finish — with a 4,096-token budget, 78 of 400 exam answers ran off the end unanswered; even at 16k, 15 still did. I scored those as their own outcome — a non-answer, not a wrong answer — because in real use a model that burns its whole budget reasoning and never responds is a third failure mode (a timeout) the system has to catch and route: raise the budget and retry, fall back to the faster non-thinking answer, or hand off to a person. How to run that at scale is the serving post’s job later in the series; the point here is only that a blank was never silently scored as right or wrong.

Did the medical tuning help? The fair way to ask is to compare a medical model against the exact base it was fine-tuned from — MedGemma vs Gemma 3, HuatuoGPT-3 vs Qwen3 — same prompt, same budget. Done that way, medical tuning was worth 0 to 4 points on chart lookup.

The images — real, but narrower than the headline. On the standard radiology VQA sets (VQA-RAD, SLAKE), MedGemma-4B beat base Gemma-4B by 21–22 points. But on a harder CT/MRI set (RadVUQA) the 4B gain shrank to about 7 points and was confined to naming which organs are present, and at 27B the medical model actually scored below its base. The image win is strongest for the small model on the standard tests and doesn’t generalize cleanly — worth saying plainly.

The exam reruns — why my first numbers were wrong. My first exam runs showed large medical wins. They were artifacts of an unfair comparison: one model thinking while the other answered blind, or a thinking model exhausting its token budget and scoring a blank. Give both the same step-by-step prompt and the same budget and the gap closes — Gemma 3 27B 241/400 against MedGemma-27B-text 238/400; Qwen 293 against HuatuoGPT-3 298. I give this its own article later in the series — the bugs I found in my own test — because most of what looked like a capability gap was a harness problem, not a real difference between the models.

Repeatability. Every question was asked at temperature 0 with a fixed seed, each in its own request — the model saw the system prompt, one patient record and one question, never an earlier question or its own previous answer. (The shared record is cached between questions about the same patient, purely to avoid re-reading it — a speed optimization, not memory: the model still answers each question on its own.) The pipeline is deterministic: the same configuration run three times produced byte-identical answers, a 0.0% flip rate, so single-run scores hold up without averaging. I also confirmed the caching isn’t shaping the result — re-running the full set with caching off, every prompt computed from scratch, moved the score by a single point out of 163 and left all but nine answers character-for-character identical (most of those nine differing only in JSON spacing). That is what lets me treat the eval like a CI check rather than a one-off demo.

Is it just my setup? Three cross-checks. I re-ran several of the models hosted on OpenRouter instead of locally; they landed within 2–3 points. I re-ran the smallest model on a newer GPU (an RTX 3060); 157 of 163 answers were identical and the score moved a single point. And I spot-checked a frontier model (Gemini 4 Argon) as a ceiling — about 96% on the questions it answered, a small but real edge over the best open model. Quantizing from Q8 to Q4 didn’t meaningfully move accuracy.

What this is not. Lookup and counting on synthetic charts, graded exactly — not diagnosis, not reasoning about a real patient, and not a verdict on any specific product. It is one task of several in this series.

This is part of a series I’m posting on LinkedIn — follow along there. Next week: the failure that worried me most wasn’t a made-up value. It was a real one, from the wrong date.