An October update to a long-horizon benchmark for AI terminal agents shows why a single leaderboard can hide important differences: GPT-6 Astra solved more of the 31 new tasks, while GPT-5.6 Sol solved slightly more of the original 46. Across all 77 tasks, Astra finished ahead, but the authors say that lead came entirely from the additions.
The benchmark, called Long-Horizon Terminal Bench (LHTB), gives AI agents extended command-line work inside software environments. Instead of answering one prompt, an agent may need to inspect files, use tools, make changes and produce an artifact that passes a hidden verifier. The October results were added to the authors' GitHub repository on Oct. 9 and the README was updated through Oct. 11. They are an author-reported follow-up to a July paper, not results contained in that earlier preprint.
The winner changes with the task set
The updated suite contains 77 tasks: the original 46 and 31 additions. Thirty of the new tasks ask an agent to reconstruct a tool's behavior from examples or observed outputs. The additions also broaden coverage into geospatial work, document and data engineering, life sciences and materials, and physical simulation, signals and control.
| Task set | GPT-6 Astra | GPT-5.6 Sol |
|---|---|---|
| All 77 tasks | 37 solved | 32 solved |
| Original 46 | 13 solved | 14 solved |
| New 31 | 24 solved | 18 solved |
“Solved” means a task earned a reward of at least 0.95; a perfect score is 1.0. On that measure, Astra's five-task lead over all 77 is accounted for by its six-task advantage on the additions, while Sol leads the older set by one. The result is a reminder that benchmark composition can change the ordering: the overall count does not mean one model led on every part of the test.
The authors also report mean rewards of 0.901 for Astra and 0.813 for Sol on the new tasks, versus 0.552 and 0.578 on the original set. Those averages should not be compared across sets: the new reconstruction tasks use a staged scoring ladder that awards partial credit. Solve counts are the clearer within-report comparison here.
How the evaluation was run and what it can show
For this evaluation, 15 models were run against all 77 tasks with the Terminus-2 agent harness. Each model-task run had a three-hour budget and one trial. A hidden verifier checked progress every 30 minutes. Under the “continue-until-timeout” setup, an agent that stopped early could be resumed after a failed check; it received only a binary rejection, not the verifier's detailed explanation.
The benchmark requires a modified version of Harbor to reproduce the published setup; the repository warns that the stock configuration ignores an important setting. Thirteen model runs used Daytona sandboxes, while two Gemini runs were contributed through Vertex AI. The results CSV contains 1,145 scored cells out of 1,155; ten missing Gemini scores are counted as zero in mean reward. Two separate policy refusals are recorded as explicit zero scores.
There was one trial per model-task pair, and the October table has not been independently replicated. The authors' July snapshot used 46 tasks, a 90-minute budget and an older harness, so it is not a controlled before-and-after comparison. These results describe performance on this particular terminal benchmark; they do not establish how reliably an agent will handle routine production work.
Sources
- LHTB 2.0 repository and results (author-maintained; October update)
- Oct. 9 results and 31-task split commit
- LHTB task dataset card
- July 2026 LHTB paper (background; does not contain the October table)
- Image: LHTB 2.0 leaderboard figure, zli12321/LHTB, Apache License 2.0.