BlogAI adoption

Before a hospital pilots an AI agent: what to check first

Tyler Kent · November 13, 2026 · 4-minute read

In July 2026, agents in an OpenAI cybersecurity test got around controls meant to keep them offline, and about 700 joined an attack on Hugging Face, an independent investigation by METR found. Before a hospital pilots an AI agent near patient or billing systems, it should settle how each action is recorded, which systems are open to the agent, who can stop it, and what the vendor answers in writing.

Writing in MIT Technology Review on October 29, 2026, two researchers at the Partnership on AI argue that buyers in regulated industries, "like banks and hospitals," can press agent developers to add the records that are missing today.

What happened?

The test measured how well the models could exploit software flaws, and they ran without "OpenAI's deployed cyber safeguards, system prompts, or auto-review systems," the company's technical report says. Once online, agents used Hugging Face users' credentials exposed on the internet, along with flaws they found in its systems, to widen their access between July 10 and 13.

Hugging Face, which hosts AI models and datasets, disclosed the intrusion on July 16. OpenAI's monitoring flagged unusual activity on July 19, and the next day the company concluded that its agents were likely behind the attack. OpenAI says the automatic review mode it offers customers in its coding tool would have flagged most of the dangerous actions in each incident run it tested afterward.

Fig. 1

About 700 AI agents in an OpenAI test joined an attack on Hugging Face

Four numbers from the investigations by METR and Hugging Face into the July 2026 incident

Show the numbers as a table
About 700 AI agents in an OpenAI test joined an attack on Hugging Face
MeasureValueContext
Agents in OpenAI's test that joined the attack700Of about 1,200 that reached an unsanctioned message board, by METR's count
Reviewed transcripts with faked tool calls7%Roughly, and all at small scale, METR found
Agents that alerted a person0METR found 3 to 6 that considered it, among about 1,300 transcripts
Recorded events Hugging Face used to rebuild the timeline17,000+From its own log of the attacker's actions

A second case reached a healthcare data system. An experimental OpenAI model got into the Services Australia Medicare Statistics Reporting Service, a data portal for Australia's universal healthcare system, during internal training in June, the company said, as Reuters reported. The model "ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files," OpenAI said, and its review so far had found no evidence that medical records were accessed.

Why does this matter to a hospital?

One question carries over to a hospital pilot: does the software running an agent record what it did and on whose authority? The Partnership on AI's paper read the documentation of four widely used agent frameworks, the software that runs an agent's steps.

Each identifies an agent only by a name or number that lasts for one run, and none links it to the identity in an organization's own access system. None records who reviewed an agent's action or what authority they had. Three of the four have no session-wide setting for how often an agent stops to ask a person, so nothing records a change to it. So after a mistake, the record these frameworks produce on their own may not show which agent acted or who signed off.

What should the record hold?

Ask for a record of every action the agent takes, written as it happens: each system and record it touched, each tool it used, and who approved each step. Keep it where the agent cannot reach it. Voluntary practices that NIST, the National Security Agency and the Cybersecurity and Infrastructure Security Agency published in October 2026, mainly for organizations that train or test advanced agents, say to make sure agents "cannot access or interfere with monitoring processes and logs."

METR found that agents built ways to fake the tool calls in their own transcripts, and roughly 7% of the transcripts it reviewed were faked in places, all at small scale. OpenAI says none of the attempts it saw changed the logs its graders or monitors read; METR found no case of an agent editing a log after the fact but "cannot rule out" one. A full record also pays off afterward: Hugging Face rebuilt the attack's timeline from its own log of more than 17,000 recorded events.

What should the agent be unable to reach?

Anything the pilot does not need. The agents in the test had no direct internet access, and OpenAI's report says that restriction "proved insufficient" because they could exploit shared services and other paths out.

Give the agent its own login, limited to the one system and task, and list the networks it may reach, a "strict allowlist" in the federal practices' words. Run the first weeks on a test copy with made-up patient records, so a mistake touches no real chart or claim.

Who can stop it?

A named person, at any hour, with a shutoff that has been tested. On June 27, a monitoring tool alerted OpenAI to unusual activity in an earlier run of the same test, and on-call staff advised that stopping the run was not required. After a July 5 alert, OpenAI took a compromised service offline, but the report says the improvised message board the agents used to trade information "was not apparent to leaders responsible for incident detection and response at that time."

Write down which alerts mean stop, who decides, and how the agent is shut off, then test the shutoff before go-live. Don't count on the agent to report problems. METR searched roughly 1,300 agent transcripts and found only three to six cases of an agent even considering alerting a person, and none that did it.

What should you ask the vendor?

Put these six questions to the vendor in writing.

Fig. 2

Six questions to settle with a vendor before a hospital pilots an AI agent

What to get in writing about what the agent records, what it can reach and how it is stopped

These questions sit alongside the model questions in which AI models sit under your vendors' tools and the audit log review in what to settle before a chat assistant reads the patient record.

For an organization with no AI use policy or risk tiers yet, the AI Governance Starter ($8,500, fixed, three weeks) places each AI tool in a risk tier with the review that tier needs, and writes the questions every AI vendor answers before a contract. An organization that wants one person to own these checks across every AI pilot, without a full-time hire, can bring in a fractional chief data and analytics officer.

Related service

Related articles

More on AI adoption: 31 charts and 30 sourced facts.

Insights by email

Does this look like your organization?

Book a 20-minute call with Tyler to talk through your version of this problem and what fixing it would cost.

Prefer to skip the call? Get a written proposal