How to score an AI vendor before the demo

A vendor's demo runs on cases the vendor picked. That tells you little about how the tool will handle your own cases. So write down what you need before you watch any demo.

Start with a requirements matrix

A requirements matrix is a one-page list of what the tool must do, drawn from the workflow itself and two or three interviews with the people who run it. Each line is ranked must, should, or nice. A prior-authorization tool for a medical group might have "reads faxed referrals as PDF" as a must, "pushes status into the EHR work queue" as a should, and "Spanish-language patient letters" as a nice.

Score every vendor against the same lines. A vendor that misses a must is out. This step alone can shorten a long shortlist.

Test the remaining vendors on your own de-identified cases: five cases from last month, run through each tool and scored by the staff who would use it. If a vendor won't run your cases, record that in the score.

Five questions about your data

Every healthcare buyer should have written answers to these five before pricing is discussed.

1. BAA. Will they sign a business associate agreement, and does it cover the AI features specifically? A BAA signed before the vendor added AI features may not cover them, so check.
2. Subprocessors. Who else touches the data? If the vendor's tool sends your data to an AI model run by another company, such as OpenAI, Google or Microsoft, that company handles your data too. Ask for the full list, the BAA chain down to that provider, and where the data is hosted.
3. Retention. How long are your prompts and outputs stored, by the vendor and by their model provider? Zero-retention and 30-day options exist. OpenAI, for one, keeps API abuse-monitoring logs for up to 30 days by default and offers zero data retention to approved customers. Treat "it depends" as a no until you get specifics.
4. Training on your data. Is your data used to train or improve their models, or the model provider's? The answer should be no, and it should be in the contract rather than a settings toggle.
5. Exit and export. When you leave, what do you get back, in what format, at what cost, and how long until your data is deleted? Ask for a deletion certificate in the contract.

Expect written answers to all five within a week.

Build versus buy, in dollars

Sometimes building is cheaper. Here is the math, with numbers invented for illustration.

For example, say the workflow is referral-packet assembly at a 200-bed hospital, and two case managers each spend about 10 hours a week on it.

Buy, in this example: $60,000 a year in license fees, $25,000 one-time integration, and 0.2 FTE of internal ownership at about $20,000 a year. Three-year cost: $180,000 plus $25,000 plus $60,000, or $265,000.

Build, in this example: a $45,000 fixed-price build, about $12,000 a year in hosting and model API costs, and 0.3 FTE of ownership at about $30,000 a year. The ownership cost is higher than for buying because your own staff has to fix whatever breaks. Three-year cost: $45,000 plus $36,000 plus $90,000, or $171,000.

On these numbers, building saves about $94,000 over three years. Building makes sense only when the workflow is narrow and stable and someone on staff will own it. If the vendor's product covers five workflows you will use, or nobody can own a build, buying is the better choice even at the higher cost.

Check one more thing. A tool's value depends on whether it saves time at the bottleneck. Twenty case-manager hours a week speed up discharges only if those hours were what held discharges up. Hours saved anywhere else free up staff time but don't move patients out any faster, and a cost comparison like the one above won't show that.

When the answer is no purchase

An evaluation can end with no purchase. Here are the reasons it can end that way.

• No vendor meets the musts.
• Every vendor fails one of the five data questions, for example because its contract hasn't caught up with what its product now does.
• The tool works on a step that isn't the bottleneck. A denial-appeal drafter does little if appeals already get filed and the delay is upstream in coding. Across the more than 2,100 hospitals and 300,000 physicians in Kodiak Solutions' data, payers initially denied nearly 12 percent of claims in 2024, so appeal volume is real, but faster appeals won't help if coding is the bottleneck.

The feature you're shopping for may also be about to arrive in software you already own. Epic shipped AI charting built into its EHR in February 2026, so anyone evaluating an ambient scribe, which drafts visit notes from the conversation, had to account for it.

Vendors can also overstate their savings or go out of business. Olive AI, an automation vendor once valued at $4 billion, announced in late 2023 that it would shut down, about a year and a half after an Axios investigation found its savings figures rested on rough estimates. So a recommendation to wait, or to build instead, is a valid result of an evaluation.

The sequence

Write the matrix, get answers to the five data questions, test on your own cases, watch the demo, then do the math. In that order, the demo confirms whether the product matches its documentation.