A low harmful-output rate does not reveal whether a model recognized a harmful request and refused it, missed the task, or answered something else. A new preprint from researchers at Shenzhen University and Harbin Institute of Technology pairs attack success rate (ASR), which measures harmful assistance, with a second, response-based measure of whether the model recognized and addressed the test task.
The authors call the second measure the operative understanding rate (UR). A response counts only if it identifies the evaluated task and treats it as the task to answer. A task-aware refusal or safe alternative counts; translating the prompt or answering a different task does not. The score comes from the response and annotation rules, not a direct reading of the model’s internal understanding.
Using the same 105 harmful tasks in both prompt formats, the researchers found GPT-4o’s ASR was 2.9% for both Toki Pona and CaesarCipher. Its operative UR was 15.2% for Toki Pona and 96.2% for CaesarCipher. The paper uses Toki Pona, a constructed language, as a semantic-compression test that leaves parts of requests implicit; CaesarCipher encodes English requests. The same ASR therefore concealed a large difference in whether replies engaged with the test task.
A high recovery rate did not guarantee a safe answer. In a separate condition, the researchers encoded the same harmful-task set as formal logic expressions rather than plain-language prompts. GPT-4o’s operative UR was 94.3%, while its ASR was 52.4%. The paired measures therefore separate whether a reply addressed the test task from whether it provided harmful assistance.
The study tested six models across seven prompt formats. When researchers rewrote the same compressed requests in three progressively clearer English versions, task recovery rose for all six models, while ASR changed differently by model. The authors caution that their curated set of 105 tasks does not show how common these cases are outside the experiment. UR depends on response annotations; it is not a direct measure of internal understanding.
Sources
- Jiang et al., “Same Outcome, Different Evidence: Intent Recovery in LLM Safety Evaluation,” arXiv preprint, submitted Oct. 8, 2026
- Full preprint, including methods, results and limitations
- Authors’ code, experiment inputs and aggregate results
- Figure 2 reproduced from the preprint under CC BY 4.0; the figure and paper are by the listed authors.