An audit of the HarmBench part of HELM Safety found that a model scoring three prompt types separately predicted pass-or-fail outcomes better than a model using one overall score. The types were direct harmful requests, longer contextual scenarios and requests to reproduce copyrighted text.
The preprint, submitted October 8, analyzes HELM Safety v1.17.0, a snapshot accessed June 21, 2026. It covers 81 models and 398 HarmBench prompts. Two AI judges scored each response on a five-point safety rubric. Only a unanimous perfect safety rating, recorded as an average of exactly 1.0, counted as a pass; any lower average counted as a failure.
To compare the models, researchers used judge-derived pass-or-fail answers to estimate refusal scores. One statistical model estimated a single overall score; another estimated separate scores for direct requests, contextual scenarios and requests to reproduce copyrighted text. They tested how well each predicted responses not used to fit it. Across five repeated splits, the three-type model had a mean held-out log-loss of 0.258, compared with 0.322 for the strongest one-factor model. Log-loss measures how closely predicted pass probabilities match held-out outcomes; lower is better. A seven-factor model scored 0.255, nearly the same, so the authors preferred the simpler three-factor version. Their results show one factor was too few for this snapshot, but do not establish an exact number of underlying traits.
The authors also checked whether OpenAI and Anthropic models with similar refusal scores still answered particular prompts differently. For the first check, they grouped models into three score bands; using the overall score, they found differences on 13 items. A second check used a continuous score instead of bands and found 17. When they repeated the checks using separate scores for direct requests, scenarios and copyrighted text, the counts fell to one and two. These are conditional, item-level differences, not a ranking of overall safety. The authors say the drop is consistent with aggregation effects, but cannot rule out real topic-specific differences between developers.
The models are a convenience sample, not a random draw, and developer groups are small: 21 OpenAI models and 11 Anthropic models. The scores depend on two AI judges and one benchmark release; the authors did not rerun HarmBench’s original classifier pipeline. The findings describe this scored snapshot, not every AI system or future version.
Sources
- Full paper, version 1
- Abstract and submission record
- Code and results repository
- Illustrative photo: Tiger Lily via Pexels; Pexels license terms checked.