Even a polished-looking computer workflow can break a user’s instruction in a small but consequential way. In a new benchmark of long desktop tasks, the strongest tested AI judge reached 80.9% balanced accuracy on the hardest set. Across the full benchmark, it correctly identified only about one in four cases involving an unwanted side effect.
AgentHorizon measures whether a judge can compare a user instruction with a recorded computer workflow and decide whether the actions complied. It evaluates the judge's ability, not the computer-using agent's ability to perform the task. The benchmark contains 1,373 instruction-and-trajectory pairs drawn from 166 hours of human-recorded work across three operating systems. Researchers created near-miss failures by pairing a workflow with a closely related but different instruction, then reviewed the resulting labels.
On the 528-item hard split, GPT-5.5 used through Codex correctly classified 71.8% of valid trajectories and 90.0% of failed ones. The reported 80.9% is the average of those two rates, giving valid and failed examples equal weight. The split was selected using model judgments, so this score is not an estimate of how often an AI agent succeeds in everyday use.
One paired benchmark example shows why a completed-looking sequence can fool a grader. The instruction asks for a WordPress post built from a specified release-notes file; the paired 195-step trajectory also adds a changelog section from a separate file. Claude Opus 4.7 judged the trajectory successful, although the authors labeled the extra material a bad side effect.
Across the full benchmark, GPT-5.5 through Codex correctly rejected and labeled 24.1% of 232 bad-side-effect examples. That exact-recall measure requires both recognizing failure and naming the right category. The paper is an October 8 arXiv preprint, not a peer-reviewed study. Its paired near-miss design does not represent all agent failures, and the authors report that independent human annotator agreement was not measured. The benchmark and evaluation code are available from the authors.
Sources
Image credit: Figure 1 from Lù et al., AgentHorizon, arXiv:2610.11050v1, CC BY 4.0.