Salesforce AI Research has introduced Critical-State RL, a method that identifies which decision in a tool-using AI conversation is worth retraining. In a controlled Berkeley Function Calling Leaderboard study, training the response made after a missing tool became available raised task accuracy from 14.0% to 28.3%, while training the earlier alternative lowered it to 9.5%.
Tool-using AI agents make a sequence of decisions: whether to wait, which function to call and what arguments to send. A final success or failure score cannot by itself show which decision caused the outcome. Reinforcement learning uses rewards to improve a model's decisions. But the same response can be followed by different later actions, so a changing reward does not necessarily reveal what that response did wrong.
Separating the action from what happens later
Critical-State RL begins with candidate calls and a local reward defined for the task. With the preceding conversation held constant, it generates several possible actions. It then holds each action constant while resampling the later interaction. Comparing the average reward across actions estimates the signal attributable to the current call; variation among continuations of the same action estimates downstream noise.
The diagnostic also checks that the local reward is connected to overall task success and that a stronger reference policy leaves room for improvement. It ranks the qualifying calls by action-dependent reward variation, then applies the training loss only to the selected response. The surrounding turns still provide context or determine the reward, but their responses do not directly contribute to the model update.
The researchers tested this choice on two BFCL v4 multi-turn situations. In a missing-function task, the required tool is withheld when the substantive request arrives and becomes available on the next turn. The model must avoid an invalid write, then recover by making the held call. In a missing-argument task, the function exists but a required argument arrives later, so the critical mistake is acting prematurely.
The useful turn changes with the failure
For no-think Gemma-4-26B-A4B, the diagnostic selected the later recovery response in the missing-function task and the earlier decision response in the missing-argument task. At a fixed step-30 checkpoint on paired 200-item evaluations, the selected recovery training improved missing-function accuracy by 14.3 percentage points. The alternative decision training reduced it by 4.5 points.
The pattern reversed for missing arguments. Training the selected decision response moved accuracy from 43.5% to 47.3%, a 3.8-point gain across four seeds. Training the recovery alternative produced a one-point increase. The authors describe the missing-argument result as directional, and the alternative cells are point estimates rather than multi-seed summaries.
A wording-change test suggests that the larger result was not limited to one scripted transition. When the message announcing renewed tool availability was replaced with an unseen, nonspecific phrase, recovery training raised accuracy from 9.5% to 22.5% on 200 paired examples. The canonical wording improved from 14.0% to 27.0% under the same single-checkpoint protocol.
The method does not automatically discover every important moment in an arbitrary agent run. The study starts from task-defined candidate phases and local labels, and its main four-cell experiment tests which category of turn to train rather than calibrating a universal diagnostic threshold.
The study is an unreviewed preprint; it does not establish a general-purpose way to find these training targets without task-specific design.
Sources
- Chen et al., “Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use,” arXiv:2609.24985 (submitted September 21, 2026)
- Full paper in arXiv HTML, including the BFCL study and limitations
- Berkeley Function Calling Leaderboard v4
- Lead image: server racks at NERSC, photographed by Derrick Coetzee and released under CC0 via Wikimedia Commons.