When many AI models reach the same score on a benchmark, that score can no longer show which one is stronger. In an Oct. 14 arXiv preprint, researchers report that comparing models’ transcripts of correctly solved tasks from saturated benchmarks predicted their performance on harder, held-out questions more accurately in a retrospective study.
A benchmark score compresses each model’s work into a number. But two systems can reach the same correct answer in different ways, and their transcripts can preserve details that the score leaves out. The researchers tested whether an AI judge could use those differences to estimate how models would perform once the benchmark’s saturated questions stopped distinguishing them.
Pairwise judging used work on questions every reference model had solved
The team defined a benchmark item as saturated when the three strongest models in a reference group all answered it correctly. For the main test, they gave evaluation methods only transcripts of correct answers to saturated items. They then tested how well each method predicted capability on harder, unsaturated items that were held back from the prediction process.
The strongest method asked a language model to compare two correct transcripts for the same task and pick the one that appeared to come from the more capable system. The researchers first used earlier models with known results on harder items to calibrate how transcript preferences related to performance. They then compared transcripts from a reference model and a newer target model to predict performance for the newer target model on those harder items. A score-only baseline could not tell the targets apart on the saturated questions, so it assigned each the reference group’s average capability.
Across 16 target models from the second half of 2025 and first half of 2026, spanning science, mathematics, coding and cybersecurity, pairwise judging had a pooled mean squared prediction error of 0.26, compared with 1.37 for the score-only baseline. The authors describe this as a 5.3-fold reduction in prediction error. That measure is error in the study’s estimated capability scale, not a 5.3-fold increase in answer accuracy.
Retrospective gains varied by domain, and extended-reasoning models exposed a calibration gap
The pooled result comes from a retrospective simulation using historical model releases, not a prospective test of a live evaluation program. When researchers examined domains separately, pairwise judging had the lowest error in all four, but its improvement over the baseline was statistically significant only in coding and cybersecurity.
The clearest failure appeared when models with extended reasoning arrived in early 2025. Judges calibrated on late-2024 systems underestimated the new reasoning models by 2.1 to 3.0 units on the paper’s capability scale. The authors say current protocols are not yet robust to such distribution shifts. Their result therefore suggests that solved-task transcripts can add a signal after scores saturate, while leaving open how reliably the method will work for future model families. Because some providers do not expose reasoning traces, the transcript information used in the study may also be unavailable for some closed models.
Sources
- Hamada, Yeon, Migliarini, Olive and Dubois, “Evaluating Model Capabilities After Benchmark Saturation,” arXiv:2610.20667, submitted Oct. 14, 2026.
- Full paper: evaluation setup, results and distribution-shift analysis.
Image credit: Figure 1 from the paper, licensed under CC BY 4.0.