An arXiv benchmark called RECLAIM asked four AI agents to reproduce a selected result from 100 NeurIPS 2025 machine-learning papers, checking whether each run’s output fell within a preset margin of the paper’s specified result. Among papers whose authors released code, data and model weights, DeepSeek-V4-Flash matched the target on 14 of 34 papers (41%), the highest rate among the four tested models for that group.
Researchers at the University of Illinois Urbana-Champaign introduced the preprint on Sept. 23. Agents received each paper and whatever artifacts its authors released, then had to install dependencies, choose how to run the experiment and document the result in execution records.
The benchmark separates papers by what their authors released. Its “Run” tier includes code, data and model weights; “Retrain” omits the weights; and “Reimplement” omits the code. Four models were assigned to each paper, creating 400 possible paper-model runs; 396 started. In the Retrain tier, DeepSeek-V4-Flash matched the selected result on 9 of 33 papers (27%). In Reimplement, where the agent must rebuild the method without the authors’ code, Muse Spark 1.2 led with 5 of 33 papers (15%).
RECLAIM grades execution evidence rather than accepting an agent’s own claim that it succeeded. Three human graders scored the 396 runs that started, one grader per run, under the same rubric and without seeing the automated auditor’s score. The most common process failure was rebuilding a method without checking it against a quantity reported in the paper, recorded in 63 of the 400 possible runs.
The authors argue GPU time was not the main limit: failed attempts used an average of 29% of their allotted GPU time. But the result does not show that the underlying papers are wrong. The authors did not run human reproductions on the same cluster, so they could not separate agent mistakes from random-seed or hardware variation in some faithful runs that missed the target. The benchmark covers one conference year and four agents, so it does not establish how all current AI assistants would perform.
Sources
- Salunkhe et al., “RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?” (arXiv preprint, Sept. 23, 2026). Figure 1 is reproduced under the paper’s CC BY 4.0 license.
- RECLAIM code, frozen benchmark and run records.
- RECLAIM benchmark dataset.