A Qwen3-32B model was tested with 16-bit and 4-bit weights on six tasks.
Average accuracy fell by 0.43 percentage points.
Layer-by-layer analysis found that later errors often opposed earlier ones.
Large language models can often reduce the precision of their stored weights without losing much accuracy. Researchers from Tsinghua University and Bosch AI Research have traced one reason inside Qwen3-32B: errors introduced in one layer are frequently pushed in the opposite direction by later layers.
The process is called post-training quantisation. Once training is complete, high-precision weight values are approximated with a smaller set of numbers. Fewer bits reduce the storage and memory traffic required for inference, but every approximation changes the calculation slightly. The puzzle is why those small differences do not simply accumulate across dozens of transformer blocks.
The researchers compared the original BF16 weights with Nvidia's four-bit floating-point format, NVFP4, on six zero-shot tasks. Attention and feed-forward linear weights were converted to four bits, while embeddings, normalisation layers, activations and the final language-model head remained at higher precision. Across the six tasks, mean accuracy declined by 0.43 percentage points.
They then passed the same text through both versions and measured the hidden-state difference after each block. In the trained model, the error created by a block tended to point against the error already present. These interactions cancelled 81.8% of the newly accumulated error. In a randomly initialised model, the corresponding figure was 0.09%. Removing the opposing component in layers 17 through 48 increased the final relative error 2.94 times; reversing it increased the error 8.41 times.
The final word scores were also less sensitive than the hidden-state movement suggested. Around 85% of the original top-10 and top-20 token candidates remained in those sets after quantisation, although the top choice changed in 8.3% to 12.7% of cases depending on the dataset.
The paper is a preprint and studies next-token prediction, not the stability of long generated answers. It also does not measure the speed or memory use of a production four-bit kernel. The result is therefore a mechanism, not a blanket performance claim. It suggests that future compression methods may benefit from finding layers where this cancellation is weak and preserving only those layers at higher precision.
Sources
Why Does Post-Training Quantization Work?
GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers
AWQ: Activation-Aware Weight Quantization