A new benchmark found that Claude Opus 5.5 reached the human expert baseline with less single-GPU time on seven AI research tasks. The result measures iterative experiment design for fixed problems, not whether a model can choose what research to pursue.
Oliver Jaffe and Dane Sherburn of P-Zero Research built TasteVal to test one part of research work: given a defined problem, can a system choose experiments, interpret their results and improve a solution? The benchmark does not score whether the system picked an important problem in the first place.
In each run, the model under evaluation acted as the Researcher. It proposed experiments and interpreted reports but could not write or run experiment code or inspect the test set. A fixed Claude Opus 4.8 agent served as Coder, implementing each proposed experiment and running it on one NVIDIA H100 GPU. The paper tests 20 models on eight tasks, with six runs per model-task pair and limits of 40 H100 GPU-hours or 120 wall-clock hours per run.
The human baseline came from 24 experts, with at least two attempting each task; researchers used the strongest expert attempt on each task as the reference. Against that baseline, Opus 5.5 in its maximum-reasoning configuration scored a 2.30 compute multiplier, with a 95% confidence interval of 1.15 to 4.37. In this metric, a 2× multiplier means reaching the same score with half the serial experimental time on the same hardware. It does not mean twice the final quality or twice as many completed research projects.
Opus 5.5 refused one task, which the researchers treated as missing data rather than a failed run. When they recalculated the comparison for all models and human experts on the same seven tasks, Opus 5.5's multiplier was 2.25, with a 95% interval of 1.16 to 4.29. That interval is wide, and task-to-task results varied substantially.
The benchmark is deliberately easier to verify than real research: tasks use clean data and quick, low-noise scoring, fit on one H100, and do not require real-time coordination. The authors say these choices may make the results unrepresentative of frontier AI development. The tasks are also withheld to reduce contamination, limiting independent reproduction from the paper.
In a separate October 2 capability disclosure, Anthropic said Opus 5.5 was not close to substituting for its research scientists and engineers on an internal test of real engineering problems. That test is different from TasteVal and its score is not directly comparable. A broader assessment will require independent replication and evaluations that include choosing problems and handling less controlled research work.
Sources
- Oliver Jaffe and Dane Sherburn, “TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts,” arXiv:2610.06824v1, October 5, 2026. Figure 2 overview reused under the paper's CC BY 4.0 license.
- Anthropic, Transparency Hub, Claude Opus 5.5 model report, October 2, 2026.