Tether AI Research has released QVAC Genesis III, a 191-billion-token collection of synthetic documents across science, technology, engineering and mathematics for training smaller language models. In the authors' test, a model trained on the full collection answered 42.71% of ARC-Challenge science questions correctly, against 21.36% for a same-size model trained on a similar token budget from Cosmopedia-v2.
The release contains about 159.6 million documents across 19 STEM domains. To make them, the team generated multiple-choice questions from filtered web text and had a smaller model try to answer. When the smaller model answered correctly, a more capable model wrote explanations for the right and wrong options. When the smaller model answered incorrectly, the more capable model wrote a corrective explanation. This turns both outcomes into material for pre-training, the stage in which a model learns from large amounts of text before being adapted to a particular task.
The larger dataset beat a repeated-data baseline
The authors trained 1.7-billion-parameter models from random initialization with the same architecture and training settings. The full Genesis III run used its 191.43 billion tokens once. To give the Cosmopedia-v2 model a similar number of training tokens, they passed through its roughly 27.5 billion unique tokens seven times. Genesis III scored 42.71% versus 21.36% on ARC-Challenge, but the gain was smaller on the harder GPQA Diamond science test: 22.72% versus 20.20%. Matching processed tokens in this way does not match the amount of distinct material seen by each model.
The public release carries error warnings
Researchers can inspect the dataset, generation code and trained model checkpoints. The dataset carries a CC BY-NC 4.0 license for noncommercial use. Its own card warns that synthetic documents can contain reasoning errors and incorrect answer keys and should not serve as factual references or validated data for high-stakes work.
.jpg){kind=link}