On a new benchmark for AI coding agents assigned to customer projects, Claude Opus 5.5 met the authors’ resolution threshold on 14 of 95 simulated tasks. Its average FDE-Gym score was 0.56, compared with 1 for an expert-authored reference solution and 0 for leaving the customer’s system unchanged.
FDE-Gym tests work that starts before a coding ticket is clear. A forward-deployed engineer is assigned to a customer to find out what the customer actually needs, check the rules around its systems and deliver a working change. In the benchmark, agents had to do that in simulated customer environments built around 20 delivery scenarios from 18 sectors.
The sponsor’s request could miss the business problem
Each task placed an agent in a workspace with a customer’s existing system, files, historical data and a staff directory. The agent could inspect or change the system and contact scripted stakeholders. It then submitted its work for deployment against held-out data. Some sponsor requests described the wrong fix or left out constraints that changed what a successful solution should do.
One drilling scenario illustrates the gap. A sponsor asked for a daily summary email and a chat box, but the underlying business problem involved disputed downtime claims: an existing script sometimes assigned downtime to the wrong party. Solving only the stated request could pass functional checks without improving the customer’s metric.
The score rewards business improvement and charges for contact
The researchers measured whether each delivered system improved the customer’s own business measure. A score of 0 meant no change from the original system; 1 matched an expert reference solution. The final score also deducted preset costs for messages to stakeholders, test runs and waiting. A production incident could cap the positive result. The contact charges were modelled by the benchmark authors, not measured from real customer projects.
Across 13 model configurations from seven providers, Claude Opus 5.5 had the highest mean score in the realistic setting, where the sponsor’s original request could omit important facts. It resolved 14 tasks, defined as scoring at least 0.8 with no incident. Five configurations resolved none. Even in the full-information setting, where Claude Opus 5.5 obtained every key fact on 23 tasks, its delivered system reached at least 0.8 of the expert reference’s business improvement on only 10 of them before contact costs were deducted.
Scripted customers leave a real-world gap
The benchmark changes how much information each task gives the agent, from a full statement of the problem and constraints to the sponsor’s original request. Supplying the rules helped the strongest configurations, but none exceeded 0.82 even with the full information. The researchers conclude that finding hidden constraints is a major bottleneck, while the remaining gap also reflects how well agents turn facts into a useful system.
FDE-Gym is a version-one preprint, and its customer environments are simulated. The stakeholder replies are scripted, the contact costs are preset, and the authors have not yet reported a human-expert baseline. Results also describe the models in one shared, minimal agent framework, not the commercial products built around them. The authors make three example cases public; other task variants and all expert reference solutions remain held out.
Sources
- Abstract and submission record, “FDE-Gym: The Last Mile of AI Delivery,” arXiv, Oct. 15, 2026
- Full paper, including methods, results and limitations
- FDE-Gym source code and documentation
- Figure 2 from Dongxu Yang et al., “FDE-Gym: The Last Mile of AI Delivery,” arXiv:2610.22350v1 (2026), CC BY 4.0; unmodified. CC BY 4.0 license.