A world model that distinguishes the effects of different moves improved simulated robot planning and a small pick-and-place test, researchers at Nanjing University report. The approach addresses a planning problem: predicting what happened is not always enough to choose what to do next.
A model-predictive-control (MPC) planner tests possible action sequences in a learned model, estimates which outcomes come closest to a visual goal, then executes part of the best sequence and plans again. If two camera views share most of their visual content, a model can predict the next view accurately while missing the smaller differences that tell the planner whether one move helps more than another.
The team’s action-discriminative world model, or AD-WM, is trained to keep those differences. Instead of predicting an entire next image representation from scratch, it predicts the change from the current representation. An auxiliary training objective also asks whether the action can be recovered from the predicted transition. The extra recovery heads are removed at deployment, leaving the MPC planner unchanged.
In the OGBench Cube simulation, AD-WM succeeded in 52.0% of trials averaged across five hard-start protocols, compared with 3.7% for a matched LeWorldModel (LeWM) baseline. The protocols began with the gripper away from the cube and varied the cube’s starting position; each used 50 episodes per model checkpoint, and results were averaged over three training seeds. On the original-distribution test, the same models scored 90.7% and 73.3%, respectively.
Across five simulated environments, AD-WM had a higher mean success rate than the reproduced LeWM baseline in four. PushT was the exception, at 92% versus 94%; the gain on the Scene benchmark was not resolved across three seeds. The results therefore support an improvement over this matched baseline, not a win across every task or over all robot-planning methods.
On one Franka robot setup, AD-WM completed 32 of 45 basic pick-and-place trials, compared with 19 of 45 for V-JEPA 2-AC. This was a separate comparison from the LeWM simulation tests: both robot models used the same frozen V-JEPA 2 vision encoder, filtered DROID manipulation data for post-training and the same planner, while the transition model and training objectives differed. The robot followed manually supplied image goals for grasping, moving and placing, and the models were tested in separate, non-randomized blocks. No Franka lab images or demonstrations were used for adaptation.
The results also show why lower prediction error alone can be a poor guide to control. On the Cube task, LeWM had lower one-step and five-step prediction error even though its hard-start success was lower. In separate fixed-candidate simulation diagnostics, measures of how good the moves retained for further search were more closely associated with success; those diagnostics did not measure live, adaptive planning.
The robot result remains a limited transfer test: one site, camera and visual backbone, with manual subgoals. The paper does not establish how well AD-WM would work across other robots or less structured tasks.
Sources
- AD-WM paper (full text)
- AD-WM project page, code and demonstrations
- Image: Shixart1985 / Wikimedia Commons, CC BY 2.0. The illustrative image is not from the study; the homepage card crops it to 16:9.
