AutoSFT searches for supervised fine-tuning (SFT) recipes: which training examples a model uses, how heavily it samples them, and whether examples are revised or added. An arXiv paper reports that, after 30 rounds, a Qwen3-1.7B-Base model trained through AutoSFT’s configuration with four analysis agents scored 67.10 on IFEval, an instruction-following benchmark. That is 6.5 points above the single-agent setup’s best score in its first 30 rounds. Both comparisons used the same base model and dataset, three training epochs, and a 3-million-token cap per round. Each configuration is one search, so repeated scoring of a checkpoint does not show how results vary across independent searches.
Figure 1 from AutoSFT by Bingxiang He and co-authors, licensed under CC BY 4.0.
Why the authors split the work
In a pilot, a single agent completed 32 evaluated rounds in 11.5 hours but kept adjusting category proportions; it did not repair examples, replace answers or write new ones. The paper says the agent’s context also filled with training and evaluation output.
Analysis agents propose changes; synthesis can add examples
AutoSFT uses a coordinating agent, an executor and specialized research agents. The executor runs training and evaluation in its own session, then returns saved results and artifact paths so long job logs do not consume the coordinator’s working context. Four analysis agents inspect different evidence: training examples, failed answers, learning signals and earlier experiments. Each suggests a data change; the coordinator decides what to try next.
A separate synthesis agent can write new examples when the pool has too few for a tested capability. Before a generated example enters the pool, it must pass a programmatic check for the target constraint, an independent semantic review and an n-gram overlap check against evaluation data. The coordinator decides whether to use the new examples for training.
Main IFEval comparison, separate synthesis and coding tests
In the controlled analysis comparison, each round retrained from the same base checkpoint. The analysis setup reached 67.10 on IFEval at round 30. The single-agent baseline’s best score in its first 30 rounds was 60.60; its best across 50 rounds was 62.35. The authors say the runs shared a time budget. Scores average 10 stochastic evaluations of one checkpoint, and the paper reports only one research search for each setup. The result therefore shows a measured gap in this comparison, not the expected gap across repeated searches.
The synthesis test was separate and used a 100-million-token cap per round. There, the best IFEval scores were 81.58 with synthesis and 78.15 without it; the best IFBench scores were 51.95 and 49.02. The larger budget and different component mean these figures should not be combined with the 3-million-token analysis result. Added examples did not solve every behavior: one tested constraint remained at zero after three synthesis attempts.
Within the shared 64-hour budget for the coding experiment, the base and analysis-only runs each completed 15 rounds, while analysis plus synthesis completed six. On MBPP+, analysis-only scored 64.8% at round 13, while analysis plus synthesis scored 65.8% at round six. Because the runs progressed at different rates, these are results from different round counts, not a same-round comparison. The coding test covers one additional domain.
Evidence remains narrow
All experiments used one base model at one scale, Qwen3-1.7B-Base, and each component comparison ran one search per setup. Averaging repeated evaluations helps measure a checkpoint; it does not measure variation between independent searches. The authors say results may differ at other model sizes. The paper links an AutoSFT GitHub repository, which currently says “Source code coming soon,” so outside teams cannot yet reproduce the implementation from that page.