Johns Hopkins researchers have proposed a way to steer language models toward answers that score better on a chosen task while leaving their trained weights unchanged. During generation, it adjusts the scores assigned to possible next tokens and penalizes changes that shift token probabilities too much. In code-generation tests, it produced a higher unit-test pass rate than best-of-N sampling—a baseline that keeps the highest-reward answer from a fixed number of samples—on two of three models and tied on the third.
The method, Minimally Invasive Steering Vector Optimization (MISVO), addresses a risk of test-time reward optimization: pushing a model toward a score can move its answers far from its original behavior. MISVO adds a separate set of adjustment values to the model’s internal representation just before the final layer scores possible next tokens. It adjusts those values at each output position while keeping model weights fixed. A penalty based on Fisher information reflects how much an adjustment changes token probabilities, not simply its size; its gradient avoids backpropagating through the transformer body.
The evaluation used four open-weight instruction models ranging from 1.2 billion to 14 billion parameters. On 500 prompts from the Stanford Human Preferences dataset, the team optimized and scored answers with the same learned Skywork model, which assigns a preference score; this shows what that scorer preferred, not how independent human raters judged the answers. For code, it tested 120 MBPP+ programming problems and measured each answer by the fraction of held-out unit tests it passed. Methods used matched numbers of generated responses. MISVO reached a 73.9% pass rate with LFM2.5-1.2B, compared with 70.3% for best-of-N sampling; on Gemma 3 4B, the figures were 76.1% and 73.9%. On Llama 3 8B, both methods scored 73.3%.
In the paper’s one-GPU profile, ten timed prompts were run one at a time after five warmups on a single NVIDIA H200 NVL, with generation capped at 512 tokens. For LFM2.5-1.2B, MISVO averaged 49.3 seconds per prompt and peaked at 10.7 gigabytes of GPU memory, compared with 24.4 seconds and 3.7 gigabytes for best-of-N. The authors did not test cross-prompt batching, which may change throughput.
The theory behind the penalty is local: for a fixed answer length, the authors show its estimate of how the probability distribution over complete answers shifts tracks the full sequence-level calculation only when steering is small. They do not show that large adjustments keep answers close to the original model. The direct divergence test uses partial sequences generated by the unmodified model, not complete answers generated with steering enabled.
Sources
- Minimally Invasive Steering of Language Models, arXiv:2609.30218v1, submitted September 24, 2026; the report includes the method, benchmark setup, results and limitations.
- Illustrative neural-network diagram by QuantuMechaniX8 via Wikimedia Commons, dedicated to the public domain under CC0 1.0; it does not depict the study’s method or experiment.
