Researchers have built GPU kernels that return identical results for key AI calculations on three generations of Nvidia hardware. In tests reported September 22, the fixed-order matrix math also ran faster than a prior method for making model outputs more reproducible.
A language model set to choose its most likely next word should give the same answer to the same prompt. Yet that answer can change when the model runs on a different GPU. The arithmetic is sensitive to the order in which a chip adds partial results: tiny rounding differences can change which word wins a close contest.
Fixing the order of GPU math
The team at Georgia Tech and the University of California, Merced targeted the model's linear layers, which perform much of its matrix multiplication. Their kernels store weights in the compact BF16 format, widen them to more precise FP32 values inside GPU registers, and add the products in a preset order. That order depends on the calculation's shape, rather than on the GPU model or the order in which parallel tasks finish.
The comparison method, LayerCast, also uses FP32 arithmetic but first creates full-size FP32 copies of the weights and then calls a device-tuned matrix-multiplication library. That library can choose different addition orders on different GPUs. The new kernels avoid those copies and move the smaller BF16 weights from memory, reducing weight traffic during decoding.
Matching layers, faster runs and one remaining mismatch
In seven fixed-input tests, every tested linear-layer output matched bit for bit across Nvidia A100, L40S and H100 GPUs; LayerCast's outputs did not match across all three on any of those shapes. In end-to-end tests using three language models and four question-answering benchmarks, the new method ran 1.17 to 3.1 times faster than LayerCast across the reported hardware and workloads. The tests used single GPUs running the same software environment.
Whole-model outputs were not perfectly identical. One of the 12 model-and-benchmark combinations still had a different token stream across GPUs on 0.08% of problems. The researchers traced the difference to an attention calculation outside their fixed-order linear layers. The team has released its kernel and evaluation code for closer testing.
Sources
- Cooper and colleagues, “Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures”, arXiv, September 22, 2026.
- Research code and evaluation scripts, GitHub.
- Illustrative image: 3D GPU workstation rendering by Wikideas1, CC0 1.0. It does not depict the research team's test hardware.
