A rare-event estimator called Iterative Unalignment measured controlled language-model events with probabilities near one in a billion by training a temporary sampling model to visit those events more often. Exact likelihood ratios then reweighted its samples back to the fixed base model, preserving the probability the researchers wanted to estimate.
Ordinary Monte Carlo sampling becomes impractical when a failure appears once in millions or billions of trajectories. Red teaming can find a path to a failure, but finding one path does not reveal how often the unchanged model would take it. Importance sampling addresses that gap by drawing from a proposal distribution that visits the event more frequently and correcting each sample for the proposal's change in probability.
The difficult part for language models is constructing a proposal that changes many context-dependent token distributions without collapsing onto one unusual route. If common event paths under the base model receive too little probability under the proposal, a few very large correction weights can dominate the estimate and make its variance explode.
The proposal changes weights, but the target model stays fixed
Iterative Unalignment, or IU, creates the proposal by adding LoRA adapters, small sets of trainable parameters, to the language model and optimizing a differentiable surrogate associated with the event. The exact binary event definition remains separate. A reverse-KL regularizer restrains the proposal from drifting too far, and its strength adapts using the dispersion of observed importance weights. After training, the method discards the optimization batches, draws fresh trajectories from the proposal and computes the base-model probability with exact reweighting.
This separation lets a dense signal guide search before any true event has appeared. It also creates a central limitation: a useful surrogate must raise the real event across its important modes, not merely one easy subset. The authors state that their observed effective-sample statistic cannot certify coverage of event paths that were never sampled.
Controlled events make the deep tail verifiable
The evaluation covered 319 events on GPT-2 Small and Gemma-2-2B across three families. Token-presence tests asked whether a generated sequence contained a specified token. Compositional tests required several tokens in an order or in any order. A third family used a profanity classifier threshold. These are tractable proxies rather than a benchmark of realistic autonomous-agent catastrophes.
For token presence, the researchers derived an unbiased low-variance reference estimator from the model's next-token probabilities. That produced reference values down to 1.08 × 10-9 on Gemma-2 and 3.97 × 10-9 on GPT-2 Small, with reported relative standard errors below 10%. The reference calculation is more efficient than counting only observed target tokens, so the one-in-a-billion results do not imply that ordinary sampling directly saw billions of outputs.
The main accuracy test asked how many samples each estimator needed for 95% of repeated estimates to fall within a factor of ten of the reference probability. On GPT-2 Small token-presence events between 10-8 and 10-7, IU's compute-weighted efficiency gain exceeded 800-fold over naive Monte Carlo at sequence lengths of 10, 30 and 50 tokens. That accounting included proposal training as well as final sampling on an isolated A100 GPU.
Sampling efficiency was larger than total-compute efficiency because each proposal had a fixed training cost. For one set of 50-token GPT-2 experiments, IU's geometric-mean estimator efficiency exceeded 20,000-fold across the displayed error thresholds, while training each proposal used 300 steps and 38,400 trajectories. The paper reports about 25 million forward passes on batches of 128 sequences for the full token-presence training and evaluation sweep.
Complex event structure reduced the gains. For ordered three-token events, raw sample efficiency reached 2,090-fold, but the best reported compute-weighted gain was 35.3-fold. When the same three tokens could appear in any order, covering multiple valid routes was harder: sample efficiency reached 196-fold and compute-weighted efficiency 11.3-fold. Performance also weakened on Gemma-2, whose vocabulary is much larger than GPT-2 Small's.
The experiments used small and midsize language models and precisely defined events with reference probabilities. They did not test realistic, semantically defined agent failures, larger frontier models or long tool-using trajectories.
Sources
- Iterative Unalignment paper record and abstract, submitted September 21, 2026
- Full paper with estimator design, benchmarks and limitations
- Official implementation
- Lead image: Figure 2 from the paper, published under its CC BY 4.0 license.