Researchers at the Chinese Academy of Sciences and collaborating institutions have tested a way to restore math performance in language models compressed to fewer than three bits per weight. They let each compressed model generate answers, then used a full-precision teacher to guide it along the paths it actually took. On one controlled MATH-500 comparison, a 1.7-billion-parameter model solved 45.6% of problems after this extra training, against 23.2% after an equal-budget extension of the existing method.
Reducing the number of bits used to store model weights can cut memory demands, but at these very low widths the model may drift during a long answer. The first recovery step, quantization-aware distillation, teaches the compressed student to match a teacher on fixed text prefixes. At use time, however, the student must continue from its own earlier tokens. A small deviation can change the context for every later token, and the paper documents repeated or unfinished responses on math tasks.
Teacher guidance on generated answers raises math scores
The researchers added on-policy distillation after that first recovery step. The student generates responses through the same quantized computation path used for deployment. A frozen, full-precision teacher scores the student's next-token choices along those responses, while task checks reward correct math answers and code that passes tests. The training updates full-precision master weights before exporting the compressed model; it does not require the full-precision teacher when the recovered model is used.
In the authors' matched comparison, both approaches started from the same checkpoint after the first recovery step and used the same training prompts and optimizer-step budget. Continuing the fixed-prefix method produced 23.2% accuracy on MATH-500, while the added on-policy step reached 45.6%. Across four models and two effective weight widths, the paper reports that the compressed models' average MATH-500 performance rose from 35% to 70% of their full-precision counterparts' scores after the new step. Those are relative retention figures, not absolute test accuracy.
Released Qwen3 checkpoints support benchmark reruns
The team released code and Qwen3 checkpoints for evaluating the compressed models and rerunning the additional training. Its repository does not include the Falcon3 checkpoint used as cross-model evidence in the paper. The tests cover fixed math, code and short-answer benchmarks rather than real deployments, and the study does not report an inference-speed or memory-use comparison for the recovered models.
