A benchmark for clinical AI agents checks whether an appointment or test order actually appears in a health record, not just whether an agent says it completed the task. Researchers described the system in a peer-reviewed paper published Oct. 10 in npj Digital Medicine. The same work first appeared as a preprint on Feb. 9, so the journal publication updates the paper's status rather than marking the first public release of its findings.
The benchmark, FHIR-AgentEval, uses 43 modular tasks involving appointment management and genetic testing. Agents work against a resettable server using HL7's Fast Healthcare Interoperability Resources (FHIR), a standard for exchanging electronic health information. For each task, the system checks both the agent's response and the data left on the server. That can separate a plausible-sounding answer from a workflow that actually changed the intended record.
To build the memory, the authors ran an agent on training tasks, then distilled its execution logs and validation feedback into reusable tips about specific FHIR resource-operation pairs and broader planning lessons. One version could consult FHIR R4 specifications while forming those notes. Across four test settings, memory improved task success and reduced strategic errors such as choosing the wrong tool or confusing resource types.
In the held-out experiment, the agents tested three variations of 22 task types that were not used to build the memory. The best configuration, with memory trained without the FHIR specifications, achieved a 60.6% task-success rate. The authors report that gains were smaller on these unfamiliar tasks.
The result is a benchmark finding, not evidence that an AI agent can safely manage patient records in a hospital. The tasks ran in a controlled, resettable environment, and the paper says agent reliability in clinical workflows remains unproven. For developers, the practical distinction is between an agent that describes a correct action and one whose changes leave the test system in the right state.
The authors have posted the benchmark environment and experiment code on GitHub.
Sources
- Peer-reviewed article in npj Digital Medicine, published Oct. 10, 2026
- Earlier preprint, posted Feb. 9, 2026
- Authors' FHIR-AgentEval benchmark and code repository
Illustrative stock photo by Pavel Danilyuk / Pexels, used under the Pexels License. The image does not depict the study.