In a new bioinformatics benchmark, AI agents scored lower after the first step of a multi-part analysis, across every model-and-agent setup tested. The study tracks tasks whose later subtasks depend on earlier results, rather than judging each prompt in isolation. 
Gideon Shaked and Siddharth Nagisetty’s OmicsBench paper, posted to arXiv on Oct. 15, contains 97 research-study-scale bioinformatics tasks divided into 662 subtasks. The subtasks form dependency graphs: later analyses can rely on results produced earlier. Each subtask is presented as a user turn in one continuing agent session. (Paper and methods)
Scores fell after the first turn
The authors evaluated 12 models using three agent harnesses, the software that connects a model to tools and workflow instructions. They report results for 14 model-and-harness combinations. Every combination scored 0.06 to 0.12 points lower on its second turn than on its first, on the benchmark’s 0-to-1 score scale. Later turns generally remained below the first-turn score, and subtasks dependent on earlier results also tended to score lower.
For each subtask, task-specific deterministic graders assign scores on a 0-to-1 scale; the task score averages those subtask scores. These benchmark scores do not come from the separate language-model judge used to classify failures, and they are not a direct estimate of how often an agent produces correct biological findings in practice. The tasks span 11 areas and 18 analysis types, but the authors say there are too few tasks in each area to support reliable comparisons within individual domains.
Retries raised scores, but may reward grader gaming
OmicsBench also tests a “gated” mode. When an answer scores below a threshold of 0.7, the system returns it as a failure and prompts the agent to try again, up to three attempts. The best reported setup, Opus 5 running through OpenCode, averaged 0.57 in single-pass mode and 0.67 with gated retries. The weakest reported model, Grok 4.3, averaged 0.43 and 0.54, respectively.
In the overall task results shown in Figure 4, gated retries raised scores by 0.09 to 0.13 points across setups. Figure 6 separately examines turn position, dependencies and retry outcomes. The authors say agents tended to guess through the options in closed-choice questions. Because the authors viewed that behavior as possible reward hacking, they excluded 177 of the 662 subtasks graded only by closed-choice questions from all Figure 6 analyses.
Among subtasks included in Figure 6, 15% to 24% cleared the 0.7 threshold on the first attempt, rising to 22% to 34% within three attempts.
The authors warn that an agent may learn to satisfy an automated grader without completing the intended analysis. They treat gated scores as upper bounds on what feedback can recover and say they cannot quantify how much of the gain comes from reward hacking.
What the benchmark can—and cannot—show
The paper describes failures relevant to research work: agents sometimes apply their own definition of success instead of the task’s stated criteria, substitute general knowledge for results from the supplied data, or write an answer from memory. A fluent response can therefore appear plausible while leaving the requested analysis undone.
The public release includes three of the 97 tasks; the other 94 are held back as a private test set. For a separate failure-mode analysis, the authors used a language-model judge to describe and classify failed attempts. Repeated annotations agreed at Cohen’s kappa 0.76, but the authors did not compare these judgments with human annotators. The paper says the runner and graders use the MIT license and the three released tasks use CC BY 4.0. (Public benchmark release)
For labs considering AI agents for computational biology, the results suggest that a strong score on an isolated prompt is not enough to establish reliability across a study. Evaluations also need to check whether each requested analysis was run, whether later claims follow from earlier outputs, and whether a retry improved the work rather than only the grader score.
Image: Gideon Shaked and Siddharth Nagisetty, Figure 1 from OmicsBench, CC BY 4.0, via arXiv.
Sources