LLM-42 checks a block of generated text by recalculating it in the same GPU operation order before returning it, helping make only designated requests repeatable while other model traffic stays on its usual fast path. Microsoft Research and University of Washington researchers are presenting the system at SOSP 2026, which opened in Prague on Sept. 29.
The same model can produce different text across runs even with the same prompt and sampling settings. In a busy serving system, dynamic batching changes which requests are processed together. That can change the order in which a GPU combines numbers; because finite-precision addition can give slightly different results when grouped in another order, the difference can carry forward into later generated tokens.
LLM-42’s fast path uses ordinary GPU kernels to propose tokens while requests are batched dynamically. The verifier then recalculates a fixed-size window using the same-shaped operations, so the GPU follows a consistent reduction order across runs. Matching tokens are released; if a token differs, the system rolls back to its last consistent point and decodes again. Only requests flagged for repeatability need this check.
In the authors’ tests, with 10% of requests marked deterministic, LLM-42 produced 33% more throughput than SGLang’s all-request deterministic mode on the ShareGPT workload. The comparison used Llama 3.1 8B Instruct on four NVIDIA H100 PCIe GPUs with SGLang 0.5.3rc0; the paper also reports synthetic and ArXiv workloads.
This remains the researchers’ evaluation of one model and setup, not an independent replication. Their prototype also pauses all in-flight requests during verification and does not batch prompt prefill, so serving latency and larger deployments still need testing. The team has made its implementation public.
Sources
- LLM-42 paper and system diagram; Figure 1 reused under CC BY 4.0.
- SOSP 2026 official conference page.
- Microsoft Research’s SOSP 2026 program.
- LLM-42 implementation and benchmark code.