In an Oct. 2 technical report, ServiceNow described AutoSynthData, a system that turns an AI agent’s failures into new training tasks and checks those tasks before using them to fine-tune the model. In two tests inside the company’s simulated EnterpriseOps Gym, the approach improved benchmark scores; ServiceNow has not reported results from live business systems.
Enterprise agents must work through changing records, tools and access rules. A model may know how to call a tool yet still fail to complete a multi-step workflow or leave the system in the wrong state. ServiceNow’s EnterpriseOps Gym is a containerized test environment for such tasks, rather than a live customer deployment.
AutoSynthData first compares a target model’s performance on diagnostic tasks with a stronger model’s successful runs. It distills the capability being tested, where the target struggles and what a correct result requires into short specifications. The task generator receives these specifications, not the original evaluation prompts, records, action sequences or scoring rules, and creates new tasks with different wording and starting states.
Each task includes the environment rules, a user request and a verifier: a check that determines whether the final system state satisfies the request. Before a sample is accepted for supervised fine-tuning, the pipeline runs a reference solution and checks that it passes; it also tests whether relevant incorrect outcomes are rejected. Failed samples can receive a limited, targeted repair and are checked again. A separate batch review looks for repetitive examples and capability gaps.
In the Hybrid part of EnterpriseOps Gym, ServiceNow used Gemma-4-26B-A4B-it as the target model and Qwen3.8-27B as the teacher. The system generated 2,000 training samples in about 18 hours. After fine-tuning, the mean Pass@1 score—a measure of task success from one attempt—rose by 7.2 percentage points, which the company reports as a 35% relative gain. On the Hybrid benchmark evaluation, its task-verifier success rate rose from 63.01% to 68.55%.
A separate ITSM test used the same target model but DeepSeek-V4.1-Flash as the teacher. It generated 1,994 samples in 66 hours, and mean Pass@1 rose from 18.77% to 27.18%. These results come from distinct simulated domains and setups, so they should not be read as one combined comparison.
The findings are company-reported improvements within EnterpriseOps Gym; the report does not establish that they transfer to customer workflows or have been independently replicated. ServiceNow presents reinforcement-learning use as future work. The linked benchmark dataset is listed under Apache-2.0, but the report does not announce a release of AutoSynthData itself or its fine-tuned model weights.
.jpg){kind=link}