In the paper’s breakdown of failed direct solutions across the tested models, 83.6% fell into cases where the model also failed the separate Discovery test for identifying that problem’s key idea. The result suggests models can stumble before the calculations begin, although the test covers a small, curated set of problems.
The team calls that key idea a mathematical primitive: a compact, problem-specific observation—such as an invariant, theorem condition or reformulation—that makes a solution possible. Their Prim benchmark separates four tasks: finding the idea from a problem, solving the problem directly, recognizing the idea after seeing a correct solution, and solving when the idea is supplied.
On Prim, Qwen3.6-27B identified the key idea on 24.73% of problems, but could solve 52.75% directly. When shown a correct solution, it identified the idea on 92.31%; when given the gold idea before solving, it reached 78.57% accuracy. Those are separate tests, so the results do not mean the model can reliably turn every supplied hint into a correct proof.
Across the 12 models, supplying the benchmark’s gold primitive raised solution accuracy by 17.58 to 29.67 percentage points. The authors also tested Absorb, a self-distillation method in which a primitive-guided teacher trains a model along its own solution attempts. The trained student does not need the primitive at inference. On three Qwen3.5 sizes, the authors report average gains of 2.42 to 4.48 points over the base models across four math benchmarks.
The study draws its diagnostic questions from 182 text-only math problems in HLE-Verified, a subset of Humanity’s Last Exam. GPT-5.4 drafted the key-idea annotations, which three experts with graduate-level mathematics training reviewed; GPT-5.4 was also used to judge whether model-generated ideas matched the expert-checked ones. The authors released the benchmark, code and trained checkpoints, but independent replication has not been established. The result is evidence about this test set, not a general measure of mathematical understanding.
For AI developers, the paper argues that final-answer scores can conceal whether a model failed to choose the right mathematical approach or failed to execute it.
Sources
- The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models (arXiv preprint, October 1, 2026; CC BY 4.0).
- Authors’ project page and code repository, with released data and checkpoints.
- Lead illustration: the authors’ project teaser, adapted from Figure 1; image hosted with the arXiv paper, CC BY 4.0.