What AI benchmarks miss in healthcare work, and how to test a tool on your own tasks

Mercor, a company that sells human-made data to AI developers and publishes its own AI benchmarks, estimates that 90% of American working hours go to tasks with little or no coverage from existing AI benchmarks, in its Benchmark Map as of October 2026. A benchmark is a set of test prompts used to score an AI model, and it is where a vendor's "benchmark score" comes from. Healthcare sits below that average: only 12 of 89 clinical and technical healthcare jobs have even one task the benchmarks test, and none of the 19 healthcare support jobs, such as home health aides, do. So test any AI tool on your own staff's tasks before you buy it.

What did Mercor measure?

Mercor needed a list of the work people do, so it used O*NET, the U.S. Department of Labor's free database of jobs. O*NET describes more than 900 occupations and lists the tasks each one involves. A registered nurse's list has 27 tasks, among them "Record patients' medical information and vital signs" and "Monitor, record, and report symptoms or changes in patients' conditions."

Mercor took the prompts from 29 AI benchmarks and had an AI model match each prompt to the O*NET tasks it fits. A task counts as covered when at least five prompts match it. By that bar, 924 of O*NET's 18,797 tasks are covered. To get the 90% figure, Mercor weighted each task by the share of working time it takes up, using estimates other researchers made, also with an AI model.

The company calls the results provisional. It counted only benchmarks that have at least 25 public tasks, that ask a model to do work rather than answer questions, and that the most advanced models don't yet ace, and it added its own APEX benchmarks. So a test that quizzes a model on medical knowledge doesn't count toward coverage here. It also capped how many benchmarks each job family could have. It says that leaving benchmarks out pushes its coverage numbers down, by an amount that is difficult to bound, so real coverage could be higher. It says it hopes to "help fill" the gaps it finds.

How much healthcare work do the benchmarks cover?

Less than the economy as a whole. In Mercor's data, 208 of all 923 occupations (23%) have at least one covered task. O*NET puts clinical jobs in a group called healthcare practitioner and technical occupations, which includes nurses, pharmacists and technicians as well as doctors. Of its 89 occupations, 12 have a covered task (13%). Among the 19 healthcare support occupations, such as home health aides and dental assistants, none do, and not one of their tasks matches even a single benchmark prompt.

The 12 healthcare practitioner and technical occupations with a covered task are 11 physician specialties plus pharmacists, and even those are thin: radiologists have 2 of their 30 O*NET tasks covered. Registered nurses have none of their 27 tasks covered, and neither do nurse practitioners, physician assistants or medical records specialists. Those zeros use Mercor's bar of five matching prompts. Ten of the 27 nursing tasks match between one and four prompts, so the benchmarks do touch nursing work, just too lightly for Mercor to count it.

The map shows a similar split in law. Benchmarks heavily cover lawyers' tasks, Mercor says, but barely cover any tasks of legal secretaries or court clerks, though it describes both groups' tasks as similarly exposed to AI.

What does that mean for a vendor's benchmark score?

A benchmark score tells you how a model did on that benchmark's tasks. If those are not your staff's tasks, the score says little about how the tool will do on your work. The company makes a related point: the gap "makes it very hard to know how good AI models are at the work Americans actually do," and measuring ability job by job is "essential" for "enterprises deciding which models fit their needs." It also says benchmarks often overstate how models perform in real-world use. Benchmarks are one kind of evidence; studies of results in real use are another, and where healthcare AI has proof, and where it does not scores seven kinds of healthcare AI on that evidence.

So when a vendor cites a benchmark score, ask four things before you weigh it:

• Which benchmark produced the score, and what tasks its prompts ask the model to do.
• Whether any of those tasks match the work you want the tool to take on, in your setting.
• Which model version was tested, and whether the product you would buy runs that same model.
• Who ran the test: the vendor, the model maker or an outside group.

If none of the benchmark's tasks match yours, the score tells you little about your work, and a test on your own tasks is the evidence that does. A score on matching tasks is still a score on someone else's cases, so it narrows the shortlist; it does not replace your test.

How do you test a tool on your own tasks?

Five steps take a test from a list of tasks to a number you can compare across vendors, and each one builds on the step before it.

For an AI scribe, the test cases can be short scripted visits with fictional patients that your clinicians act out, as the questions on the AI Vendor Evaluation page describe, because a recording of a real visit is patient data. For other tools, how to score an AI vendor before the demo pairs the test with a requirements matrix, a one-page list of what the tool must do. Score the vendors against the matrix first. Then run five de-identified cases from last month through each tool still in the running, and have the staff who would use it score the results.

Record how long each output takes to fix, so you can set it against any time savings the vendor claims.

Who can run the test for you?

The AI Vendor Evaluation runs the vendors on your shortlist through the same test cases: de-identified cases from your organization's work or, for a scribe, scripted visits. It scores each vendor on those cases and on its security and HIPAA terms, the work to connect it to your systems and its three-year cost. It takes two weeks and costs $7,500 for up to three vendors or $9,500 for four or five.