Researchers at Illinois Institute of Technology introduced CertID in an Oct. 2 preprint, a benchmark that tests whether reasoning models know when a causal effect cannot be identified from a graph and observational data. Across its test set, the models’ rates of asserting answers to provably unanswerable questions differed 17-fold, despite an overall-accuracy spread of about 10 percentage points.
A causal graph lays out which factors are assumed to affect others. If an unmeasured factor influences both a treatment and an outcome, an observed association may not reveal what the treatment itself changed. For some graphs, no amount of additional data on the same observed variables can determine the exact intervention effect without more assumptions or measurements.
CertID is designed to test that boundary. It uses a formal identification algorithm to certify whether an effect can be recovered from the graph. When a model supplies an effect formula, the researchers also check it numerically against structural causal models with known intervention outcomes, rather than grading by whether the wording matches one reference answer.
The team tested Gemini Flash, Gemini Pro and GPT-5.5 on 1,200 certified cases, with graphs ranging from four to 50 variables. The models reached 97% to 100% accuracy at deciding identifiability on graphs generated after the strongest model’s training snapshot. Yet on the same unidentifiable queries, their false-claim rates varied 17-fold. A high overall score therefore did not show how often a system would still assert an effect when the graph ruled one out.
The result is limited to a formal task where the graph and question are supplied. It does not show that a model can discover the right causal structure from real-world evidence or decide whether the graph’s assumptions are credible. The paper is a preprint, and it tests three proprietary model endpoints. The authors have released the benchmark, verifier and per-case records, giving other evaluators a way to measure unsupported causal answers separately from accuracy.
Sources
- Behnam and Wang, “Reasoning Models Are Accurate but Unsound on Identification” (arXiv:2610.03519v1), submitted Oct. 2, 2026
- CertID benchmark, verifier and per-instance records
- Illustration: Marcos M. López de Prado via Wikimedia Commons, CC0 1.0. The explanatory diagram is not from the study.
_and_after_(right)_control.png){kind=link}