Global Tech News For people figuring out what to build with AI.
AI

A learned policy helps an LLM choose when to adapt on a curated test stream

By ·

Diagram of a generic neural network with connected nodes arranged in layers.

Researchers at Singapore’s NUS AI Institute and National University of Singapore trained a policy to decide when a language model should adapt as new questions arrive. In a fixed benchmark stream, it raised the model’s score from 0.256 to 0.510. The test set was deliberately enriched with questions where adaptation had shown benefits, so the result does not establish expected gains on ordinary query streams.

Test-time training (TTT) updates a model using signals from inputs seen during evaluation or deployment. A method that suits one task can improve answers, while a poor match can waste computing time or lower answer quality. Agentic-TTT is designed to choose whether to adapt, which method to use, or whether a previously built adaptation can help.

The pretrained backbone stays frozen, while researchers train the policy using its own low-rank adaptation (LoRA) adapter. LoRA adapters are small sets of parameters that change behavior without replacing the original weights. The policy can answer immediately, load an earlier TTT skill, or save a query and its accompanying material in a pool. When a pool contains material suitable for a chosen method, one of five TTT methods trains a separate skill adapter for reuse on later questions. Some methods can learn from one bundled document; others combine compatible queries.

The policy is trained in two stages. For each training query, heuristic rules propose a sequence of actions; the researchers keep a proposed action sequence only when a separate run confirms its expected benefit. The initial policy then practices choosing those actions with hints explaining what to do and why. The authors keep examples only when the policy’s chosen action matches the accepted target, remove the hints, and train on the policy’s own reasoning and the accepted actions. Reinforcement learning then tunes decisions using answer outcomes while penalizing extra computation.

At evaluation, the system processed a fixed stream with 176 held-out scored questions and 19 unscored document-ingestion events. The skill library started empty. With Qwen2.5-7B-Instruct as its backbone, overall utility rose from 0.256 for direct answering to 0.510 for the learned policy. Utility is the average of binary scores from task-specific automatic verifiers across eight domains and 11 benchmark families; the 0.510 result is the mean across three training runs, with a standard deviation of 0.012. The 19 document-ingestion events do not enter that score. Nor does the utility score subtract computation cost; the paper accounts for cost separately during policy training and in a separate compute trade-off experiment.

The benchmark’s composition matters. About 65% of its curated questions had shown an adaptation benefit during screening, a deliberate departure from the original source distributions. The authors say the results characterize this selected setting and do not establish a consistent advantage across arbitrary deployment workloads.

The training setup also leaves a long-term question open. During reinforcement learning, researchers advanced the stream using annotated target actions, while the policy made decisions from the resulting states. In deployment, the policy’s own earlier choices would create those later states. The authors say this mismatch can expose the policy to different states than it will encounter autonomously. They also note that reinforcement learning scores decisions query by query, so it does not directly teach whether collecting material now is worthwhile because a skill may improve an answer much later. The paper is an arXiv version 1; its results are limited to the reported model and benchmark experiments.

Sources