A robot can see a ball and still miss how far it will roll if the ball carries a hidden load. A new method called T2-z uses the result of one or two exploratory robot trajectories to adapt a video-predicting model to hidden conditions such as friction, weight or latch stiffness. The authors report higher action success in four lab tasks while keeping the model's learned weights fixed.
The model learns a compact code for each environment
A robot world model predicts what may happen after an action. If two scenes look alike but behave differently, the prediction can be wrong unless the model accounts for the hidden difference. T2-z gives the model a small environment code and trains the two parts in alternating turns: first the model learns shared motion patterns across environments, then the code adjusts to explain how one environment responds. The study builds on a video model called BWM, based on Wan2.2-TI2V-5B.
At test time, T2-z freezes the trained model. It starts a new code for an unfamiliar setup and tunes only that code against the observed results of one or two interaction trajectories. It can then use the adapted model to compare actions it has not tried in that setup. A trajectory here is a sequence of robot actions and the resulting observations, not a single movement.
Four physical tasks show higher action success
In tests with a Franka robot arm, the researchers changed ball loads, door-spring stiffness, stick weight distribution and the behavior of a soft object. The arithmetic mean of the four task-specific action-success rates rose from 26.47% with the standard video model, which did not adapt to the test environment, to 70.80% with T2-z. Mean tracked-state prediction error fell from 19.30 to 10.85 pixels. These are averages across four different tasks, not a rate pooled over every test video.
The gains were not identical on every measure. In the soft-object task, T2-z selected goal-reaching actions more often, but its image-quality scores were slightly worse than the standard model's. The paper evaluates seven simulated settings as well as the four physical tasks.
The test setup sets the limit
The physical tests used a lab robot and a fixed camera. For action selection, each candidate set included at least one action that met the task goal, so the results show how well the models chose among those options rather than whether they can plan any task from scratch. The paper is a version-one arXiv preprint; the reported results have not been independently replicated.
Sources
Image: Figure 1 from the T2-z paper by Chenyuan Zhou and coauthors, licensed under CC BY 4.0 via arXiv.