Researchers at the Technical University of Munich and Helmholtz Munich found that detailed task guidance affected a coding agent’s scores more than changing its underlying model size in three of four scientific tasks. The new arXiv preprint tests a benchmark spanning astrophysics and genomics.
A specialist model is software built for a particular scientific job. The agent had to decide whether a specialist could help, then find and run it by writing code. Tasks included estimating galaxy redshifts from images, classifying DNA sequences and predicting RNA structure. In one astronomy task, the specialist was weaker than the general-purpose model, so the agent needed to recognize when not to use it.
The researchers varied five parts of the setup: how much task-specific information the agent received, whether it kept its reasoning across steps, how strongly a prompt asked it to verify answers, how much time it had, and the size of its underlying language model. The main experiment tested 432 combinations for each task and repeated every combination five times, for 8,640 runs.
Those repeated trials showed why one score can mislead. On the three tasks where the specialist model outperformed the base model, repeating the same setup accounted for about 54% of score variation among runs that cleared the task’s minimum-performance threshold. A single run therefore could not reliably show whether one configuration was better.
Across those same three tasks, the amount of information supplied had the largest measured effect on score. The researchers tested four levels, from naming the specialist model to giving the agent a complete recipe for using it; information ranked first on all three tasks. More detailed guidance also reduced runtime and cost and brought the agent’s predicted score closer to the score it actually achieved. Extra time helped mainly when the agent already had enough information or model capacity to use it; on the redshift task, extending time without operating instructions lowered scores.
Asking the agent to check its own work had little effect. In separate experiments with two models, a tool that scored each answer against the task’s reference result raised scores on every task and reduced overconfidence, but also increased runtime and cost. The authors describe this reference-scoring tool as a strong form of verification, more powerful than an ordinary production validator.
The study covers four tasks in two scientific fields and does not vary components such as memory or multi-agent coordination. The authors report 18,240 released execution traces: the total includes the 8,640-run main grid plus additional model and reference-scoring experiments. The data are linked publicly, while the authors’ repository says the full code and benchmark are still forthcoming.
Sources
- Wiedmann et al., “Agents Are Systems, Not Models: Rethinking Agentic Evaluation”, arXiv:2610.01618v1, posted 1 October 2026. Figure 1 reused under the paper’s CC BY 4.0 license.
- Authors’ repository, currently listing the full code and benchmark as forthcoming.
- Authors’ released agent-trajectory dataset.