A compressed language model can use different numerical formats for reading a prompt and generating its reply, rather than forcing both stages to share the same weights. Researchers at ISTA and Nvidia tested that split with a Qwen model: a more precise prompt-processing copy improved scores in one evaluation, while a separate local test delivered the first response token faster for long prompts. Short prompts did not get the same speed benefit.
The two stages put different demands on a computer. Prefill processes the input prompt and builds a cache of information the model will need later. Decode then generates tokens one at a time using that cache. Prefill can be limited by computation on long inputs; decode repeatedly reads model weights from memory. Compressing the weights helps the latter fit, but very low-bit weights can damage the information extracted from the prompt.
Separate weights for the prompt and the reply
The researchers call their approach disaggregated quantization. They kept an existing one-bit decoder and trained a separate four-bit prefiller for the Qwen3.8-27B model. In their offloaded implementation, the computer loads the extra prefiller weights from storage for prompt processing, then uses the smaller resident decoder for generation. That avoids keeping both weight sets in accelerator memory at once, though loading the prefiller adds a fixed cost.
First-token waits fell on long prompts; accuracy was tested separately
In a modified llama.cpp run on a DGX Spark, the time to first token for an 8,000-token prompt fell from 12.27 seconds with the one-bit model alone to 6.90 seconds with the offloaded prefiller. Across tested prompts from 4,000 to 32,000 tokens, first-token speedups ranged from 1.38 to 1.78 times. The storage load made shorter prompts slower. These timings followed a warm-up and used three repetitions with prompt caching disabled.
The accuracy comparison used a different setup. Researchers dequantized the fixed one-bit GGUF decoder checkpoint to BF16 for evaluation in vLLM on a GB300; the test did not serve its weights in their packed one-bit form. Each format received one full benchmark evaluation. On MMLU-Pro, that checkpoint scored 29.04% when its weights handled both stages and 61.54% with the separate four-bit prefiller; on MMMU-Pro, scores rose from 24.39% to 59.65%. Those are gains from pairing a prefiller with this decoder, not a speed-and-accuracy result measured together in one serving configuration. Gains were smaller with a two-bit decoder, and a three-bit decoder could lose accuracy with the separate prefiller.
The study did not test highly batched requests, multi-turn conversations or agent-style workloads. Those conditions could change the cost of loading prompt weights and reusing cached information.
Sources
- Disaggregated Quantization: Specializing LLM Prefill and Decode, ISTA and Nvidia preprint, 22 September 2026, including experiments and limitations.
- Authors' modified llama.cpp implementation.
- Lead illustration: Nvidia DGX-B200-HGX by Pokiiri, CC BY-SA 4.0, used without modification. The pictured accelerator is illustrative, not the hardware used for the reported measurements.
