A new AI coding agent reran analyses from 20 published studies that used data from the US National Cancer Institute’s SEER cancer-surveillance program. It reproduced 1,694 of 2,190 individual reported results, according to a new preprint. That 77.4% score counts individual claims, not whole papers, and does not show that the studies’ medical conclusions are correct.
A benchmark built around cancer-registry studies
The researchers created SEER-Repro to test whether large language model agents could help reproduce published medical research. They assembled 10 of the most-cited eligible SEER studies and 10 more recent papers, alongside data and documentation. The agent used the papers and supporting materials to write and run code that reran the analyses, then compared its output with 2,190 atomic claims, or individual reported results.
The agent reproduced 1,694 claims, or 77.4%. Another 76 (3.5%) were classified as close, while 420 (19.2%) were not reproduced. A claim-level score is not a pass rate for the 20 papers: each paper contributes multiple claims, and one paper can contain both reproduced and unreproduced results.
The researchers’ error analysis separates data problems from mistakes by the agent. They assigned 251 of the 420 unreproduced claims (59.8%) to data issues, including unavailable inputs and mismatches involving revised records or population denominators. They classified 48 (11.4%) as agent-side errors. Other cases involved analysis descriptions that did not specify one unique method, unavailable software, suspected evaluation errors, or an unresolved cause.
That distinction matters. A failure to reproduce a result can reflect missing historical data or an unclear method, rather than a wrong original finding. But the agent’s own errors show why an automated rerun cannot be treated as a verdict.
Some results were flagged for expert review
The authors also tested whether findings changed under reasonable alternative analysis choices. This separate check covered only 579 findings the team had fully reproduced. It flagged 55 of them (9.5%) as potentially affected and sent them for expert review.
A co-author with medical expertise and experience with SEER manually examined 10 of the flagged cases. Nine did not establish a consequential change under a defensible alternative; one remained open for closer manuscript review and statistical consultation. Because only 10 of 55 were examined, and the reviewer was a co-author, this was not an independent external audit of the full set.
The authors describe the pipeline as a way to give researchers executable baselines and focus attention on analyses that may deserve closer inspection. They also warn that the system can make errors in data selection, code, interpretation, or grading, and that sensitivity to an alternative does not by itself invalidate a study.
For now, the result is a bounded demonstration on 20 studies using one cancer-registry data source, not an estimate of AI reproducibility across medicine and not clinical guidance. Its most practical promise is triage: automate some reconstruction work, show the evidence behind a mismatch, and leave judgments about scientific meaning to people who can inspect the data and methods.
Sources
- “AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery” (arXiv:2610.23845)
- National Cancer Institute Visuals Online: “Chemical Lab, Cancer Research Labs 1931”; image AV-3100-4299, photographer unknown, source credited to G. Terry Sharrer, Ph.D., National Museum of American History; public domain.