Small batches of large-language-model requests used only 5.86% of the row capacity in an Nvidia H100's matrix instructions on average, according to measurements by researchers at KTH Royal Institute of Technology. The same instructions were fully occupied while processing a fresh prompt. The September 11 preprint shows why a single headline measure of GPU utilisation can hide the part of the chip that is actually idle.
Language-model inference has two distinct stages. During prefill, the accelerator processes many prompt tokens together. During decode, it produces one new token for each active request and repeats the operation. If only a few requests are being served at once, the matrix presented to the hardware can be much shorter. The number of simultaneous requests is the batch size.
The H100 instruction examined in the study works on fragments that are 64 rows high. A decode batch of one fills only one row, or 1.56% of the fragment; a batch of eight fills 12.5%. The unused positions perform no useful token work. Because the empty rows arise in the fixed hardware instruction, merely rearranging the same small matrix in software cannot fill them.
The researchers profiled four language models with the vLLM serving system on one H100 NVL accelerator. They measured cold prompts, cached prompts and token generation across 104 combinations of context length and batch size. The main decode tests used batches from one to eight, followed by batch-32 checks where memory allowed.
For Qwen3-14B, raising the batch from one to 32 increased active compute coverage to 71.9% and cut the time per generated token from 10.10 to 2.38 milliseconds. Memory dependencies nevertheless caused most decode stalls in the small-batch tests. Larger batches improve throughput, but they consume more memory and may make an individual request wait while the server gathers other work.
The measurements cover one H100 model, one software stack and 16-bit arithmetic on a single GPU. They do not represent live traffic or newer accelerators. Operators comparing inference systems therefore need separate figures for batch size, token latency, matrix fill and memory stalls; a device-wide utilisation percentage alone cannot show where capacity is being lost.
Related coverage
- AI Inference Split Raises Tokens per Joule by Up to 56% in GPU Proxy. A related inference study tests whether separating prompt and token generation can improve energy efficiency.
Sources
- Mohammad Siavashi and colleagues, arXiv, September 11, 2026. Abstract, authorship, submission date and principal findings.
- Full paper. Hardware setup, profiler definitions, model grid, measurements and limitations.
- Lead image: Geekerwan, Wikimedia Commons, licensed under CC BY 3.0. The image is illustrative and does not show the reported experiment.
_004.png){kind=link}