A new preprint reports that GPT-6 Astra, the top model in its test, produced a mathematically correct and readable diagram on 31.25% of 512 advanced-math tasks on its first attempt. The result comes from MathVis, a benchmark that checks more than whether a generated picture can be rendered.
A rendered diagram can still get the mathematics wrong
MathVis asks a model to turn a written mathematical statement into a figure encoded as TikZ or SVG, formats that describe drawings as editable instructions. Its 512 tasks span 27 sub-disciplines, including topology, complex analysis and differential geometry.
The benchmark evaluates each figure in stages. First, its code must parse or compile and produce a non-empty image. Then the figure must preserve the mathematical objects and relationships in the prompt, such as which region a point lies in or which direction an arrow points. It also has to be readable. A polished diagram that reverses a mapping or obscures a required label does not pass.
To score those requirements, the authors compare each output with a structured reference of expected objects and relationships, plus task-specific checks. The model does not see the reference figure. This lets the test separate a rendering failure from a mathematical or visual one.
Format and rendering success and first-try full success are separate scores
On the first attempt for each task, GPT-6 Astra passed every requirement on 31.25% of prompts, or 160 of 512. Claude Fable 5 passed 26.56%, and Gemini 3.1 Pro Preview passed 22.85%. These are results on this frozen question set, not a general ranking of the models.
A separate technical check passed 94.40% of GPT-6 Astra’s 1,536 generated outputs across three attempts per task. That check only asks whether the output has the right format, can be parsed or compiled, renders successfully and is non-empty. It does not say the diagram is mathematically correct or easy to read. The 94.40% figure counts three attempts per task; the 31.25% figure counts only the first attempt on each of 512 tasks.
The benchmark does not show that diagrams improve reasoning
The authors also report a small set of agent-assisted repair examples, but say the extra steps did not reliably fix mistakes in mathematical meaning. Those examples are separate from the 512-task direct-model scorecard.
The results are preliminary: the work is an arXiv preprint, and the paper reports point estimates without uncertainty intervals. It says detailed question-level logs are retained for only three of the 16 model configurations and that some run settings and expert-calibration statistics are not reported. The study also does not establish that diagrams cause better mathematical reasoning or improve student outcomes.