Microsoft says its diagnostic AI beat doctors on hard cases: what a hospital should ask before trusting a claim like that

Microsoft's diagnostic AI system, MAI-DxO, reached the right diagnosis in 83.2% of hard cases from the New England Journal of Medicine, against 36.1% for generalist physicians, in a study Microsoft's researchers published in Science on October 22, 2026. The cases ran in a simulation, the doctors worked alone, and how much of the result came from Microsoft's own system depended on which AI model ran inside it.

TechCrunch reported the claim the same day, and all 15 of the paper's authors list Microsoft as their affiliation. A hospital that hears a vendor cite a study like this should know what to ask first.

What did Microsoft test?

Microsoft built a benchmark, a fixed set of cases for scoring AI systems, and calls it the Sequential Diagnosis Benchmark. It has two sets of cases. The hard set is 304 case records from the New England Journal of Medicine. An earlier version of the study notes that these cases are "selected for difficulty and instructional potential," so its authors were not surprised that physicians found them difficult. The second set comes from 488 emergency department visits that ended in a hospital admission, drawn from MIMIC-IV, a research database of de-identified hospital records, and the published results use 325 of them.

Each case starts from a short summary. The doctor or the AI then asks questions and orders tests, and each test adds an estimated price to the case's cost. In Microsoft's published code, an AI model answers from the case record and returns "a synthetic result" when the record lacks the one asked for. A second AI model then scores the final diagnosis on a 1-to-5 scale, and a 4 or 5 counts as correct. In the 2025 version of the study, Microsoft's own physicians graded a sample of the diagnoses and agreed strongly with the AI grader. MAI-DxO itself, in Microsoft's words, "operates like a panel of virtual physicians" that propose diagnoses, choose tests and weigh each test's value against its cost.

Who were the doctors, and what could they use?

They were generalists, doctors who treat a wide range of conditions rather than one specialty, and each worked alone. Microsoft's announcement says the physicians scored 36% on the hard cases when they could use textbooks and the internet, and 22% without them. Neither the announcement nor the abstract says how many doctors took part. In the earlier version Microsoft posted online in 2025, 21 doctors (17 primary care physicians and four hospital generalists in the US or UK, with a median of 12 years of experience) worked the 56 most recent hard cases between them, without search engines or other outside resources, and that version had no emergency cases. Its authors wrote that in practice these doctors "might refer more complex cases to specialists."

How much of the result is Microsoft's system?

It depends on the model and the cases. On the 325 emergency cases, OpenAI's GPT-5 with only a basic prompt got 76.6% right, and the same model inside MAI-DxO got 80.3%, according to Microsoft's results table and its notes on the dataset. The physicians, with reference materials, got 46.8%. So GPT-5 alone was already 29.8 points ahead of them, and MAI-DxO added 3.7 more. The abstract doesn't say how many of these cases the physicians worked, so the results may not cover the same cases. On these cases the average estimated test cost was higher inside MAI-DxO, about $4,980 a case against $4,240.

On the hard cases, Microsoft's results table gives GPT-5 alone 78.9% and MAI-DxO 83.2%, and there MAI-DxO cut the average estimated test cost from $8,409 a case to $5,431. TechCrunch, reporting on the paper, gives results for GPT-5.5, a newer model. With GPT-5.5 inside it, MAI-DxO lifted the hard-case score from 72% to nearly 87%, but on the emergency cases GPT-5.5 scored about 85% with or without MAI-DxO. Microsoft says the gains "persisted" and "improved" when it repeated the test with newer models.

So part of an AI product's result can come from the model it runs on, and a buyer should ask which model that is and what the product adds to it.

Has it been tested with real patients?

Not in this study. The abstract says the findings "were obtained in a controlled benchmark setting," and Microsoft's dataset notes say the benchmark "does not assess treatment safety, longitudinal management, or patient outcomes." Microsoft says its next step is a study with Beth Israel Deaconess Medical Center in Boston, where patients will use the system and its performance will be measured against physicians' independent assessments.

What should a hospital ask a vendor that cites a study like this?

These six questions come from what this study did and did not test, and they apply to any vendor's study.

A study on someone else's cases can narrow a shortlist, and what AI benchmarks miss in healthcare work sets out five steps to test a tool on your own. Where healthcare AI has proof, and where it does not scores seven kinds of healthcare AI on the evidence from real use.

Who can run that test for you?

The AI Vendor Evaluation tests each vendor on your shortlist with cases from your own organization's work, de-identified or fictional, and scores each one on those cases, its security and HIPAA terms, the work to connect it to your systems and its three-year cost. It takes two weeks and costs $7,500 for up to three vendors or $9,500 for four or five.