When a closed AI model gives an answer, its API may offer no useful signal about whether that answer is right. Researchers have built Pinocchio, a separate model that estimates answer correctness from the question and response without reading the answering model's weights or token probabilities. In a held-out test, it ranked correct answers above incorrect ones more reliably than other one-pass methods tested in the paper.
Pinocchio reads a question, the target model's answer and, where available, the target's identity. For image questions it also reads the image. It outputs a correctness score in one forward pass rather than repeatedly asking the target model to answer the same question. The researchers trained it on labeled responses from seven language models across 20 benchmarks.
The score separates correct and incorrect answers
On 1,953 held-out responses to questions kept separate from training, Pinocchio scored 0.863 on AUROC, a measure of how often correct answers rank above incorrect ones across possible thresholds. The strongest comparable one-pass baseline, which combined a model's stated confidence with response length, scored 0.649 on the same responses. The comparison is internal to the paper's benchmark and tested models; it does not mean Pinocchio can certify an individual answer as true.
The researchers also applied the trained calibrator to answers from 13 models absent from its training data. It averaged 0.814 AUROC across those targets. The experiment tests whether the same checker transfers to new models; it does not establish equal performance across providers, languages or subjects.
A ranking is not a safety verdict
The model was trained on English-language benchmark responses with answers that can be graded. The paper reports weaker performance on its hardest expert-level tasks, and the authors say the method needs labeled correctness data for training. Its score could help decide which answers deserve review, but these tests do not validate use without human review in medicine, finance or other high-stakes settings. That would require testing on the actual task and checking how often errors escape a chosen review threshold.
Sources
- Pinocchio research paper, revised September 22, 2026.
- Full paper, method, matched comparison and limitations.
- Illustrative desktop image, unrelated to the experiments: Baap8969, Wikimedia Commons original, CC0 1.0; unmodified 1280-pixel rendition.
