A model’s estimate of a user’s intent can affect whether it refuses a harmful request, a study found. In one controlled test, researchers changed Llama 3.1 8B’s estimate while leaving the prompt unchanged; its refusal rate on StrongREJECT prompts fell from 98% to 62%. The intervention required direct access to the model’s internal state.
The team calls its method Belief Self-Distillation (BSD). It compresses what a model has inferred about a user from conversation into a 128-number vector, a compact internal signal researchers can inspect and edit. Two trained projections read and write the signal; the underlying language model remains frozen.
The researchers tested BSD on Llama 3.1 8B, Qwen3 8B and OLMo 3 7B. Across all three, the vector’s macro-F1 scores for predicting 13 user attributes were within 3–4 percentage points of scores from each model’s full hidden state; macro-F1 is the paper’s measure of attribute-classification quality. In a separate steering test, researchers swept the same range of intervention strengths: BSD shifted 78% of target-attribute predictions in Llama toward the requested value, compared with 21% using the raw hidden state and 13% for a random direction. These are attribute-answer flip rates, not refusals.
For the safety experiment, the researchers shifted Llama’s inferred user intent from adversarial toward benign. Refusal on StrongREJECT harmful-request prompts dropped from 98% to 62%. Editing the unrelated gender estimate left refusal at 87%, the same as the average for magnitude-matched random edits. The requests themselves stayed unchanged; refusals fell but did not disappear.
The result shows that a model’s inferred view of a user can influence refusal, but it is not a way for ordinary users to bypass a chatbot’s policy. The intervention required direct access to each model’s internal state and a projector trained for that model. The method covers 13 researcher-selected attributes, which may be inferred inaccurately or stereotyped, and the paper reports refusal rates rather than harmful completions.
Sources
- User Model Extraction via Belief Self-Distillation, arXiv:2609.31603 (v1 submitted Sept. 25, 2026).
- Authors’ code and trained projectors.
- Figure 1 by Ali Holmov and coauthors, from the paper above; CC BY 4.0.