Carnegie Mellon University researchers have introduced a robot world-action model that achieved 83.3% average success across 90 real-world trials of three two-arm manipulation tasks. ModAR completed stacking cups, folding a crumpled towel, and placing an object in a drawer more often than two models tested with the same robot platform: a unified future-and-action generator reached 66.7%, while an action-only policy reached 52.2%. The September 15 preprint shows how predicting compact descriptions of motion, object identity and depth before choosing an action can help a robot learn from videos that contain no recorded control commands.
A world-action model predicts both what the robot should do and how the observed scene may change. Many current systems represent that future as an RGB image, allowing video-generation training to contribute to robot control. Reconstructing colour and appearance, however, does not explicitly emphasize the geometry and motion needed for manipulation. The new study asks whether a model can use several more structured views of the future and whether the order in which it generates them affects the eventual action.
ModAR begins with the current camera observation, the robot's joint configuration and a task label. It predicts point tracks that describe how selected image locations move, then DINO features that encode objects and scene structure, followed by a depth map. Each completed prediction becomes context for the next, and the model generates a 16-step robot action only after those future descriptions are available. Simulation tests found no consistent gain from adding future RGB to the other three representations, so the real-world system omitted RGB generation.
The controlled simulation covered six RoboTwin tasks, including moving bottles, stacking bowls and turning a switch. Every model received 50 action-labeled robot demonstrations per task; the researchers then raised the total to 250 or 1,250 by adding demonstrations without actions. Each trained model faced 50 held-out starting conditions for every task. At the 250-demonstration scale, ModAR averaged 75% success, compared with 67% for simultaneous unified generation, 55% for disjoint future and action prediction, and 34% for a variant trained with independently sampled noise levels. Expanding the total from 50 to 1,250 demonstrations raised ModAR from 66% to 76%, while the unified model moved from 63% to 64%.
The physical tests used two YAM robot arms and 30 rollouts per model on each of the three tabletop tasks. The researchers collected 100 teleoperated robot demonstrations per task. In a separate ModAR data comparison, that robot-only set produced 70.0% average success; adding 200 in-domain human demonstrations raised it to 81.1%, and adding 1,000 related EgoDex human demonstrations raised it to 83.3%. Those human videos supervised the predicted futures rather than the robot actions, which is the route by which actionless footage contributed to control.
The study also compared ModAR with Flex-pi, a video-pretrained world-action model, on the six simulated tasks. A 30.1-million-parameter ModAR averaged 75% success against 72% for the 6-billion-parameter system while using about 20 times fewer training floating-point operations, but the authors describe this as a system-level comparison because the models differ in pretraining, scale, targets and training recipe. Sequential generation also costs time: producing four future representations and actions took 147.9 milliseconds on one RTX 5090. The experiments use three physical tasks and fixed task labels, so broader tests must show whether the method generalizes to new objects, scenes and language instructions without losing its data and compute advantages.
Related coverage
Sources
- Adam Hung and colleagues, “Modality-Autoregressive World-Action Models,” arXiv, September 15, 2026. Paper record, authorship, submission date and abstract.
- Full paper. Architecture, simulation and physical-test conditions, comparison tables, compute estimate and limitations.
- ModAR project page. Author-provided task demonstrations and method overview.
- Lead image: National Institute of Allergy and Infectious Diseases, Wikimedia Commons, public domain. The photograph is illustrative and does not show the reported experiments.
