A replay-buffer method helped a smaller Qwen3 model reach the paper’s reported math-test score after about 10 newly generated response batches. The score, Avg@16, averages benchmark results across 16 generated answers. Methods that trained only on the latest batch needed roughly 30 to more than 50 batches, depending on their update count, to reach their own reported saturated level on AIME24 and AIME25, according to a September 28 preprint.
On-policy distillation (OPD) trains a smaller “student” model with guidance from a larger “teacher” on responses the student generated itself. At each step in an answer, the teacher gives a probability for each possible next text piece. Training adjusts the student so its probabilities move closer to the teacher’s. The usual approach needs fresh responses as the student changes and sets older ones aside.
Least-Square Policy Distillation (LSPD) changes how those responses are used. Its replay-buffer variant, LSPD-RB, stores earlier prompt-and-response pairs and samples them again for training. The student matches its token probabilities to the teacher’s while an entropy term helps preserve variety in possible answers.
The paper reports results across six math benchmarks and three Qwen3 teacher–student pairs. In a closer look at training efficiency, a 4-billion-parameter Qwen3-4B teacher guided a 1.7-billion-parameter Qwen3-1.7B-Base student on AMC23, AIME24 and AIME25. Each new rollout batch contained 64 prompts and four generated responses per prompt. LSPD-RB’s Avg@16 curve leveled off after about 10 batches. Methods that reused only the latest batch took about 30 on AIME24 and AIME25 with at least four optimizer updates per batch, and more than 50 with one update. The paper calls the end levels “saturated performance” but gives no numeric cutoff for saturation.
That comparison counts newly generated batches, not total computing time. LSPD-RB ran 256 optimizer updates after each new batch, while the latest-batch methods were tested with one, four, 16 or 64 updates. The results suggest replay can reduce fresh sampling in these tests, but do not show a reduction in overall compute or training cost. The findings remain limited to a single preprint’s model setups and math benchmarks.
