A new method called LeapQuant cuts the storage needed for the recurrent state in linear-attention model layers by about 70%, according to a September 29 arXiv paper. In the authors’ 8-bit tests, it matched the average accuracy of a 32-bit baseline across 12 model-task pairs; offline inference runs also delivered higher output throughput on two Nvidia GPUs.
Linear-attention layers keep a fixed-size state that summarizes earlier tokens, rather than storing a growing key-value record for every token. But during generation, each layer still has to read, update and write that state for every new token. The repeated transfers can consume GPU memory bandwidth, while compressing and restoring the state after every update can make rounding errors accumulate.
LeapQuant stores the state in 8-bit form and keeps that compressed version fixed across a window of 16 tokens. It buffers higher-precision updates inside the window, uses them with the fixed state to calculate each output, then reconstructs and quantizes the state once at the boundary. That means the method does not skip updates; it reduces how often the full state is rounded.
The paper also uses four high-precision correction terms, called “Compensator Tokens,” to capture the state’s largest outlier patterns before the remaining values are quantized. They are parts of the state calculation, not extra words generated by the model. A smoothing step further reduces error. The authors say LeapQuant needs neither retraining nor calibration data.
With those components, each stored state uses 1.19 bytes per value, including scales and correction terms, instead of 4 bytes for FP32—a 3.4-times reduction. Across 12 model-task pairs using Qwen3.5-9B, Qwen3.5-35B-A3B and Kimi-Linear-48B-A3B-Instruct, the 8-bit method averaged 75.5% accuracy, matching the FP32 baseline; the authors averaged results over three random seeds.
In a separate prefix-caching test, which kept one state checkpoint per 1,024 cached tokens, the authors report 41% to 56% lower end-to-end GPU memory use across those three models. In offline inference against vLLM’s FP32-state baseline, output throughput was 1.23–1.60 times higher on an Nvidia B200 and 1.31–1.65 times higher on an RTX PRO 6000. The Kimi workload lengths were simulated from Qwen3.5-9B’s distribution. These are results from one arXiv preprint, not a live-service test or independent replication.
Sources
- LeapQuant paper and abstract
- Full paper, methods and evaluation
- Earlier paper introducing recurrent linear attention
- Figure 2 by Yi Pan et al., under CC BY 4.0; the diagram explains the update method, not benchmark results.