A new benchmark asked seven model-and-tool setups to work on 82 unresolved questions in mathematics and theoretical physics. GPT-6 Astra’s submissions had the highest mean solve rate, 14.02%, as judged by four AI evaluators; the authors say expert review is still needed before any answer can count as a verified solution.
OpenProblemBench draws its questions from research papers. Each one gives the model background, assumptions and previous progress, along with a goal whose decisive mathematical or computational claims the researchers considered comparatively checkable. The set contains 46 math questions and 36 in theoretical physics.
Unlike a test with a known answer key, an open problem has no reference solution for scoring. Four evaluator models therefore assessed each submitted argument against the question’s requirements. They checked assumptions, the steps in a proof and supporting calculations, then marked the result solved, partly solved or unsolved.
Across the 82 questions, GPT-6 Astra’s average judged solve rate was 14.02%, the highest of the seven tested configurations and the leader under each evaluator. The full-size open models scored 5.5% to 6.7% on average, while the Flash models scored 2.4% to 3.7%. These are judgments about submitted arguments, not proof that researchers have verified new solutions.
The comparison also combines models with different software. GPT-6 Astra and GPT-5.6 Sol ran through Codex CLI, other systems used different agent harnesses, and the study reports results for those model-and-software configurations rather than model weights alone. Each configuration submitted once per problem; a run could last up to 12 hours, with no fixed token budget and varying resource use.
The authors say independent review by subject experts is needed to establish whether a submission is correct and novel. The four evaluators could share mistakes, so agreement among them does not show how often they produce false positives or miss valid solutions. The question set is uneven across subfields, and the paper includes no human-researcher baseline.
Sources
- OpenProblemBench paper, arXiv:2610.11118v1, submitted October 8, 2026.
- Illustrative photo by Roman Mager, Wikimedia Commons, under CC0 1.0; not an image from the study.
.jpg){kind=link}