AI-written GPU kernels can pass code tests and still fail after they are inserted into an AI framework. In a new benchmark built from 31 real kernel changes, five tested language models averaged a 43.23% full-test pass rate, but only 18.71% average task success under stricter checks for model accuracy and end-to-end speed, researchers report.
The findings come from RealisticTritonBench, a study in the ASE 2026 conference proceedings. Its arXiv version first appeared on August 12; ASE says the proceedings were published October 7. The conference program lists the paper for a Tuesday presentation in Munich.
The benchmark follows real kernel changes into AI frameworks
A GPU kernel is a small program that performs an operation across many data elements in parallel. Triton gives developers a Python-like way to write such kernels for AI workloads. The researchers turned 31 merged Triton-related changes from PyTorch, vLLM and SGLang into coding tasks, spanning optimization, modification and new-kernel work.
For each task, an AI coding agent used repository context to produce a kernel, which replaced the existing implementation in a prepared framework environment. The team then checked the kernel's unit tests, whether it changed the framework model's numerical behavior, and its end-to-end inference latency. Their test setup used five language models, a bash-only mini-SWE-agent scaffold, one server with eight NVIDIA RTX 3090 GPUs, and three runs per evaluation.
Passing code checks was not the full success condition
The paper uses two unit-test measures: full-test pass rate is the share of tasks that pass every unit test, while unit-test pass rate is the share of individual tests passed. A task counted as successful only if the generated patch matched the unit-test pass rate of the reference implementation from the original merged pull request (the paper's “gold patch”), preserved model-level numerical robustness, and avoided a meaningful end-to-end latency regression. The timing ratios compare the generated kernel with that reference under the same workload; both had to be at least 0.98, allowing up to a 2% slowdown. Time to first token is the delay before the first response token; time per output token is the delay for each later generated token.
Across five models, the authors report an average full-test pass rate of 43.23% of tasks and a unit-test pass rate of 60.33% of individual tests. Only 18.71% of task attempts met the stricter overall success criteria. A kernel can pass all tests yet still change model outputs or slow inference after integration. The best model in this benchmark reached 25.81% task success, still on the same 31-task sample.
The result is a focused benchmark, not a production failure rate
RealisticTritonBench tests selected changes from three open-source frameworks on a single eight-GPU server; it does not measure how often AI-written kernels fail across deployed systems. The study's useful contribution is a reproducible path from code generation to framework-level checks, showing why kernel evaluations need to test the surrounding model and serving behavior as well as the code itself.
Sources
- RealisticTritonBench paper (arXiv; version 1, August 12)
- ASE 2026 conference program and proceedings publication date
- Image: Fleshas (BalticServers.com), Wikimedia Commons, CC BY-SA 3.0. Illustrative server room, not the paper's test server.
