A new arXiv preprint finds that language-model agents often fail to follow plans they declare. In three of four test environments, only 22% to 45% of runs matched all scorable plan steps in order; on ALFWorld, automatic routing reached 72% success against 48% for Plan+ReAct.
The study compares a ReAct loop, which chooses each next action from the environment’s feedback, with a system that first asks the model to declare a planning approach. In the baseline, the model receives its plan as context, but the loop does not require its actions to follow that plan. Planning-as-Routing uses a deterministic router to send the declared approach to a dedicated executor built to carry it out.
Three language models were tested in four benchmark environments covering web navigation, software tasks and simulated household activity. A rule-based checker required every scorable plan step to appear in the action trace in the declared order. The 22%–45% range applies to ALFWorld, Mind2Web and SWE-bench Verified; WebArena had higher structure-preservation rates.
Search, the best fixed pattern for ALFWorld, reached 92% success when used for every task. The authors selected it as the strongest fixed strategy for that benchmark; this is not an automatic-routing result. The gap leaves a separate bottleneck: whether the model can choose which strategy to use.
The September 29 paper is an arXiv preprint tested in four benchmark environments, not a live deployment. Its checker measures whether an action trace maintained a plan’s structure, not whether each action was correct. On 100 ALFWorld traces, two annotators agreed in 71 cases on whether the declared structure was preserved, showing that this judgment can itself be ambiguous. The authors say the code and trajectories will be released upon publication.
Sources
The authors’ preprint
Research project page
Image: Figure 1 by Subba Reddy Oota et al., licensed under CC BY-SA 4.0.