A single coding-agent session with access to files and code matched or beat four machine-learning engineering systems in benchmark comparisons using the stronger models tested, a Sept. 30 preprint reports. Researchers held model, hardware and time budgets constant to see whether extra software orchestration improved results.
Machine-learning engineering (MLE) tasks ask an AI system to inspect data, build and run a model, then submit predictions for scoring. A “harness” is software around the model that directs experiments, provides tools or coordinates agents. The lean baseline in this study was not a bare chatbot: it could read and write files, run code and debug errors.
Kirill Brilliantov, Alejandro Hernández-Cano and Emmanuel Abbé compared one long-running coding-agent session, called Malena, with four open-source systems: MLEvolve, AiScientist, Arbor and ScienceFlow. All methods used the same model backbones, hardware and time budgets.
On 30 selected MLE-bench tasks, the researchers measured the share of runs that earned any competition medal and each submission’s percentile against human leaderboard entries. They report that Malena matched or beat the four harnesses with every tested frontier model. The smaller Gemma 4 31B was the exception: MLEvolve had a higher mean percentile, while the medal-rate confidence intervals overlapped.
A second test used NatureBench, 40 open scientific-research tasks scored by whether a run surpassed a published result. It included only AiScientist and MLEvolve as comparators. Malena’s estimate was higher than AiScientist’s with GLM 5.2 (21.7% versus 17.5%), but slightly lower with Kimi K3 (26.7% versus 27.1%); MLEvolve scored lower in both comparisons.
The authors also tested which design choices drove performance. Direct access to working files and code execution produced the clearest gain over a text-only chat setup. Added search strategies and multi-agent coordination did not produce a statistically significant improvement in the tested coding-agent conditions.
These are benchmark results, not a verdict on how research teams should build agents. The study is an arXiv preprint and covers selected tasks and model families; it does not establish workplace reliability or cost. The authors say coordination among capable agents remains an open question.
For teams building research agents, the result is a reason to test whether added orchestration improves outcomes under the same model, compute and time limits. Whether this pattern holds in other scientific or workplace workflows remains to be measured.
Sources
- How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering? (arXiv preprint)
- OpenAI MLE-bench repository
- NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Image credit: Photo by Bernd Dittrich via Unsplash. Illustrative workstation image; it does not depict the study.