Lewis Mitchell of Adelaide University tested a text-only filter for synthetic training data that models generate from earlier versions of themselves. After six fine-tuning rounds with Llama-3.1-8B-Instruct, held-out documents from the filtered chain had 42% more unique three-word sequences than documents from the unfiltered chain. The gain is in wording diversity, not general task performance.
The paper addresses model collapse: rare wording patterns can disappear and repetition can grow as models learn from their own synthetic outputs. Mitchell applies the Kontoyiannis entropy-rate estimator to each document. It looks for repeated word sequences: longer matches point to more repetition, while shorter matches signal less-repeated phrasing. The scoring step needs no model forward pass, API call or GPU; the fine-tuning experiment itself still used a Llama model.
The main test compared three parallel six-generation fine-tuning chains built on Llama-3.1-8B-Instruct. Each chain generated 80 documents across four domains at each generation, then selected 40 for training under one of three conditions: unfiltered, ranked by the model's token log-probabilities, or ranked by text-only entropy. Generation-zero documents were shared across conditions. The generation-six comparison used 80 held-out documents per condition.
In the generation-six held-out documents, the text-only-filtered chain had 42% more distinct three-word sequences, a 30% larger vocabulary and 19% less repeated four-word sequences than the unfiltered chain. All three differences were statistically significant (p<0.001). The log-probability filter showed no significant improvement on these measures (p>0.23).
The text-only arm's held-out documents were also more similar to one another on Self-BLEU-4 (0.086 versus 0.075; p<0.0001), a small collection-level trade-off against the gains within individual documents.
A GPT-4o-mini judge scored instruction-following slightly higher for the text-only-filtered documents than for unfiltered documents (3.23 versus 3.00 out of five; p=0.020). The coherence difference was not statistically significant (2.35 versus 2.20; p=0.058), so the judge results do not establish a broad quality improvement.
HellaSwag accuracy fell from 72.9% at generation zero to roughly 65.2-65.7% after six rounds in all three conditions. MMLU and GSM8K showed no significant difference between conditions at generation six, and the paper reports a similar diversity pattern in a smaller Llama-3.2-3B run. That second test is part of the same preprint, not independent replication. The results support a text-diversity filter, not a stand-alone safeguard for model capability or factuality.
Sources
- Lewis Mitchell, No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse (arXiv preprint, submitted October 1, 2026; Figure 1 is licensed under CC BY 4.0).
- Research code and data repository, linked by the paper.
- Creative Commons Attribution 4.0 license for the paper figure.