A new benchmark of scientific judgment found that four of five tested models more often chose an AI research agent’s original plan over a senior researcher’s correction. The result comes from SciPIvot, a 248-case benchmark built from six months of AI-assisted work on 26 research projects. Its authors describe the cases as a selected challenge set, not a measure of scientific research as a whole.
Research agents can keep code and experiments running for weeks. The harder question is whether they can tell when the next experiment will answer the scientific question, or when the current result points to a faulty measurement or assumption. SciPIvot tests that narrower judgment: choosing between an AI agent’s original plan and a senior researcher’s correction.
The test reconstructs decisions from research records
The cases span six scientific fields. For each one, the researchers froze a project record from before a human intervention. A model saw that record—including code, experiment logs and progress notes—alongside two plans: the original plan proposed by the research agent and a corrected alternative reconstructed from the researcher’s intervention and later investigation. The model had to choose which plan was more likely to advance the project. It did not see the later outcome that helped establish the reference answer.
This setup separates scientific judgment from writing a plan or running an experiment. It asks whether a model can recognize the better next step when evidence and an alternative are already available.
Most models missed the correction in the selected cases
At medium reasoning effort, with the project record available, four of the five tested models scored below 50%—the chance level for choosing between two plans. The average was 39.2%. GPT-6 Astra scored 59.5% in that setting and reached 63.5% at its best tested effort level. The paper also reports a 73.4% baseline from 13 PhD students and postdoctoral researchers who used an agent assistant; that separate group is useful context, but it is not a clean head-to-head comparison with the models.
The corrections often addressed a mistaken premise rather than a shortage of computing power. In one example, after a small language model received zero scores on benchmark tasks, the agent proposed switching to a larger model. A senior researcher first checked the scoring code and found that it had failed to extract answers. Changing models would have left the measurement bug in place.
More project context did not improve this benchmark score
With the record, the models averaged 39.2% at medium effort; when given only the two plans, they averaged 55.8%. The authors say this comparison covers the whole project-reading process, including retrieval, interpretation and the interface used to inspect a project. It therefore does not isolate a general effect of adding context. In one case, a model treated an agent’s mistaken progress note as evidence that a scoring problem had already been ruled out.
The result is specific to the cases the team retained. During construction, the researchers selected decisions where model answers varied, making SciPIvot a diagnostic challenge set rather than a representative sample of research work. It tests recognition between two plans, not whether agents can generate sound research ideas or carry out experiments successfully.
The public SciPIvot repository provides the 248 questions, prompts, scoring code and aggregate results. The frozen project records behind the cases are not yet included; the paper says they are too large to upload and will be hosted separately. The study is an arXiv preprint submitted October 15, 2026.
Sources
- SciPIvot paper, arXiv:2610.21082
- SciPIvot questions, prompts, code and results
- Visual: Figure 1 from the paper, licensed CC BY 4.0 (original diagram).