A new NVIDIA paper reports that an extra training stage helped two Nemotron coding models score higher on software-engineering benchmarks after the same fine-tuning. The gains are promising, but the study is limited to NVIDIA’s own model family and does not compare the approach with spending the same computing budget on other training stages.
Teaching models to work with whole projects
The October 15 preprint examines “midtraining,” a stage placed between pretraining, when a model learns broad patterns from large datasets, and supervised fine-tuning, which trains it on examples of desired behavior. NVIDIA’s researchers use long software artifacts such as repositories, pull requests and build logs, mixed with a smaller share of agent traces that show models using tools over multiple steps.
Ordinary pretraining often gives a model only a short stretch of text at once. Whole software projects and their change histories can exceed that span, making it harder to learn how code in one file depends on code elsewhere.
The recipe lets the model read up to 256,000 tokens, the pieces of text a model processes, at once during training. In its 500-billion-token run, agent traces make up 10% of the blend, while 20% reuses the model’s original pretraining data. The traces train the model to predict its own tool-use steps while leaving returned files and logs out of what it must predict.
Models scored higher after the same fine-tuning
SWE-bench Verified tests whether an agent can resolve software issues screened by human reviewers. DeepSWE uses newly written, longer engineering tasks, with checks designed to test the requested functionality.
After the same full fine-tuning, Nemotron 3.5 Lightning’s pass@1 score on SWE-bench Verified rose from 60.7% without midtraining to 64.5% after 500 billion midtraining tokens. For the larger Nemotron 3.5 Super, 280 billion midtraining tokens raised the reported DeepSWE score from 11.7% to 15.7% and SWE-bench Verified from 72.9% to 74.3%.
Here, pass@1 is the average success rate per attempt. The paper averages three attempts per SWE-bench task and four runs for DeepSWE; it does not count a task as solved simply because any retry succeeds.
For Super, the paper estimates run-to-run variation of about two points on DeepSWE and one point on SWE-bench Verified. By those estimates, the four-point DeepSWE increase is larger than the expected variation, while the 1.4-point SWE-bench change is close to it. Each comparison is a single training run.
What the study leaves open
The results cover two Nemotron models, mainly on coding-agent tasks. The study does not test whether the recipe transfers to other model families, and it does not compare midtraining with using the same token budget for longer pretraining or supervised fine-tuning. It therefore shows that the added stage helped in these experiments, not that it is the most efficient choice for every model.
Correction, October 19, 2026: An earlier version described pass@1 as the share of tasks solved on the first attempt. This study reports the mean success rate across multiple attempts. The reported scores are unchanged.
Sources
- arXiv abstract: Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe, submitted October 15, 2026.
- Full paper, especially sections 3–6, tables 3 and 6, and Appendix C.1.
- Lead image: NVIDIA logo via Wikimedia Commons, Apache License 2.0. The logo identifies the company and does not depict the experiments.
