A reanalysis of METR’s software-task data finds that equal tenfold jumps on its AI time-horizon chart can correspond to different changes in estimated task difficulty. A rise from three to 30 minutes maps to a much smaller difficulty change than a rise from 30 minutes to five hours.
METR’s 50% time horizon is the estimated human completion time of software tasks at which a fitted curve predicts an AI system will succeed half the time. It measures task difficulty, not how long an AI can work autonomously. Drew T. Nguyen and William Fithian of UC Berkeley reanalyzed outcomes from 228 tasks and 26 AI systems; they did not run a new evaluation.
METR’s method fits a logistic curve for each AI, linking human task duration to its predicted success rate. Nguyen and Fithian use a shared-slope version, previously tested in METR research, as their baseline. They compare that fit with a monotone spline, which lets the relationship between human time and AI difficulty bend, and an item-response model, which estimates task difficulty and AI ability together.
Under the item-response model, the estimated link between human completion time and AI task difficulty is nearly flat from two to 30 minutes, then rises more steadily elsewhere. The authors say this makes a three-to-30-minute horizon rise a much smaller estimated difficulty change than a 30-minute-to-five-hour rise. The difficulty scale is relative and model-estimated, not a directly measured quantity, so the paper describes the curve’s shape rather than an absolute amount for either change.
The researchers scored how well each model predicted task success on task families left out of fitting, using five folds across 79 families. When scores were pooled across systems, both alternatives scored better than the baseline in all 12 combinations of four scoring rules and three ways of weighting the tasks. Most individual-AI uncertainty intervals, however, were too wide to establish separate improvements.
The authors say their reanalysis leaves METR’s rising time horizons intact; it refines how those estimates map to task difficulty.
The study uses one software-task benchmark, with tasks mainly in software engineering, machine learning and cybersecurity. It does not establish whether the same relationship holds in other fields or in future suites with longer tasks. Nguyen and Fithian posted the paper on arXiv on Oct. 8, 2026.
Sources
Full paper; METR time-horizon definition and methodology; METR’s March analysis of modeling assumptions.
Image credit: Kabuttey, “Data visualization,” CC BY 4.0 via Wikimedia Commons. This illustrative image is not a METR figure or study data.
