Across 25 models, 35.3% of replies that met every crisis-recognition criterion still missed at least one step for directing users to human help. DistressBench tested the models on 718 clinician-authored English conversations about suicide and self-harm; the best model met 88.4% of weighted criteria, while the median met 71.4%.
Content note: This report discusses conversations about suicide and self-harm.
Each conversation came with a checklist of actions clinicians judged important for that scenario. The 6,901 criteria across seven dimensions each carried a weight from 1 to 20. A model's score was the share of the total checklist weight earned by the actions its reply satisfied. Clinicians wrote the simulated user turns and notes; a language model drafted the rubrics and reference replies, which clinicians then revised.
Researchers ran each model three times on every case and averaged those runs for the headline scores. In multi-turn conversations, judges scored the model's final reply. A panel of three language models graded each criterion. On 75 calibration tasks, the three-model grading panel showed substantial agreement with clinician consensus after accounting for agreement expected by chance (Cohen's kappa: 0.732; 0 is chance-level agreement and 1 is perfect agreement). This checks the judges' reliability, not the evaluated models' crisis-support performance. The benchmark's reference replies scored 98.7%, but they were drafted with the criteria in view, so that figure is a ceiling check rather than a like-for-like comparison.
The 35.3% finding is conditional: it covers replies that met every crisis-identification criterion, not all benchmark responses. For steps to reduce distress (de-escalation), the median model met 46.0% of weighted criteria, compared with 74.6% for the strongest.
DistressBench v1 covers English conversations and U.S. crisis resources. Its scenarios are scripted, with no real crisis transcripts or patient records. The benchmark does not show how a deployed chatbot responds to real users or whether its replies improve outcomes. The authors describe DistressBench as an evaluation instrument, not a clinical device.
Image credit: Operator icon by Juhele, via Wikimedia Commons, CC0 1.0. The illustration is conceptual and does not depict the study.
