In an Oct. 13 report, Axiomatic AI says Lemma checks a paper’s mathematics, proofs, claims and references, and shows the validation steps behind its findings. On a SPOT subset of 26 mathematics and computer-science papers containing 29 author-confirmed equation-or-proof errors, Axiomatic says Lemma found 26 (89.7%); each paper was later corrected or withdrawn. Its results table lists 89.7% for a three-run consensus and 93.1% for the best run, defining both as shares of known errors found. The post does not explain how it combined the runs, so readers cannot reproduce the consensus calculation from the disclosed method. Google separately reports 89.7% for its Paper Assistant Tool on the same subset; the tests were separate, not a head-to-head.
SPOT is a public benchmark of 83 published papers paired with 91 author-confirmed errors serious enough to prompt an erratum or retraction, across 10 fields. Axiomatic tested only equation and proof errors in the mathematics and computer-science papers, a narrow slice of the benchmark.
Google researchers report the same 89.7% for Paper Assistant Tool, or PAT, on the 26 papers. Their paper describes an automated language-model grader audited by one of its authors and says its logic-equivalence scoring differs from SPOT’s original strict keyword matching. Axiomatic’s post does not give enough detail to align scoring procedures or reproduce its result. The matching figures do not establish that the tools perform alike on new papers.
Axiomatic also reports that Lemma found 84% of known errors in an unpublished set of 50 physics papers; Gemini 3.1 Pro found 52%. The company says the errors were confirmed against the authors’ accounts, but the papers and paper-level outcomes are not public. On a separate check of 20 papers with no known errors and later versions with fixed errors, it says Lemma raised fewer issues per page on the clean journal papers and did not flag the corrected errors.
For references, Lemma gave the right verdict on 103 of 104 examples with known answers. In a larger scan of about 28,000 references across 459 recent preprints, 98.7% matched the published record. That set had no answer key, so the percentage is a match rate, not an accuracy measure.
The tests are company-reported and have not been independently reproduced here. Public SPOT papers may have appeared in model training; Axiomatic says it found no sign of recall but cannot rule it out. The company says Lemma does not replace peer review or certify that a paper is correct.
Sources
- Axiomatic AI: Testing how well Lemma catches errors in scientific papers
- Original SPOT benchmark paper: When AI Co-Scientists Fail
- Google's PAT paper: Towards Automating Scientific Review with Google's Paper Assistant Tool
- Illustrative photograph by Alessandro Ventura via Wikimedia Commons, CC BY 4.0; unchanged. The image is not from Lemma or the tests.