Nvidia researchers have introduced a benchmark that checks how AI agents work through computer tasks, not just whether they produce the right final file. OSWorld-Pro breaks 305 tasks into 2,814 dependent milestones, making it possible to spot a missed click or a long detour that a final-result score alone would not explain.
A computer-use agent sees a desktop through screenshots and acts through a graphical interface. In the earlier OSWorld benchmark, a task is usually judged by its final output. An agent that gets almost to the end and one that fails immediately can both receive a zero. OSWorld-Pro instead tracks whether each step targets, advances or completes a milestone that must be met before the next one.
Milestones make unfinished work visible
The team collected 67,264 human labels for agent actions across the new tasks. The tasks cover unfamiliar applications, workflows spanning at least four applications, and different Linux distributions and desktop interfaces. They require an average of 3.45 applications per task, compared with 1.34 in the original OSWorld set. A language-model judge evaluated against the human annotations then scores agent trajectories at the milestone level.
The resulting measure is the share of tasks on which an agent completes every milestone that human annotators marked feasible. It is a different test from OSWorld's final-output score, so percentages from the two task sets should not be read as a controlled before-and-after change in a model's capability.
Some agents click poorly; others waste steps
In the researchers' runs, the Minimax M3 agent frequently missed the intended on-screen click target. Only 39% of its qualifying click actions advanced a milestone, versus 72% to 93% for the stronger agents in the study. A different failure appeared in a Claude Opus 5 run: the agent completed its milestones but spent 59 steps on actions unrelated to its current goal. Both cases show why a final task result can hide distinct problems that call for different fixes.
The grading method has its own limit. Against the human labels, the study's chosen AI judge was much stronger at identifying completed milestones than at deciding whether one was feasible: its macro-F1 scores were 94.3% and 61.9%, respectively. The authors therefore use human feasibility labels when computing task completion. OSWorld-Pro is a research preprint and its reported model results are not an independent audit of commercial agents.
