An AI-written software patch can pass other checks and still fail when an inference server loads a model and answers a request. In the new SWE-Serve benchmark, researchers rescored 627 agent-created patches across 19 tasks after removing only the end-to-end model-serving tests. The pass rate rose from 45.9% to 69.4%, revealing failures that the remaining tests did not catch.
SWE-Serve draws 53 engineering tasks from changes merged into SGLang, an open-source system for serving language models. The tasks cover work such as adding model support, changing decoding and fixing serving APIs. Each gives an AI coding agent a version of the repository to modify, then checks its patch against hidden tests. Nineteen tasks include tests that launch a standalone server with a real model and check its behavior through a public request or evaluation interface.
Model-serving tests expose failures other checks miss
For those 19 tasks, the researchers held the 627 patches fixed and rescored them without the model-serving tests. Passing rose from 45.9% with the complete verifier to 69.4% without those tests. In a separate control covering 12 tasks, removing end-to-end tests created about twice as many newly passing patches as removing a matched set of other tests.
One task asked agents to support a Gemma 4 mixture-of-experts model in SGLang. Sixteen of 33 patches passed every test other than the end-to-end serving tests, yet failed when the model was served through the system's public interface. The other checks had missed a failure in the complete request-to-output path.
Serving tests cannot establish deployment or maintainability
The benchmark runs tasks in sandboxes on a CPU or one H100 GPU. Its end-to-end checks exercise an actual serving process, but they are still tests, not evidence that an agent's patch was deployed for users. The paper does not test multi-GPU or multi-node work, and passing its checks cannot establish maintainability or replace maintainer review. The reported gap is specific to these SGLang-based tasks and the tested agent configurations.
Sources
- SWE-Serve research paper (September 22, 2026).
- Full paper, task design and results.
- Illustrative data-center image, unrelated to the test system: Charlie fong, Wikimedia Commons original, CC BY-SA 4.0; unmodified 1280-pixel rendition.
