Three versions of Cosmos 3, trained on as much as 30,000 hours of first-person footage, predicted hand motion more faithfully than the objects people handled, researchers report. The difference persisted on test footage from different people, locations and cameras, raising a practical question for video world models meant to simulate how actions change a scene.
A world model is a video predictor that estimates how a scene will change after an action. Robotics researchers hope such models can provide virtual scenes for training and planning, but they are useful only if they predict both an agent’s movements and the objects’ responses.
Researchers affiliated with the University of Illinois Urbana-Champaign, Carnegie Mellon University, Johns Hopkins University and Toyota Research Institute trained several Cosmos 3 video-model variants on about 1.15 million first-person clips. The dataset covered 30,012 hours of everyday human manipulation from 14,019 contributors. Given one frame and a sequence of arm and hand movements, the model predicted the next 16 frames, or 1.6 seconds. Evaluation footage came from different people, locations and cameras than the training set.
The team scored hands and manipulated objects separately with a structural consistency score (SCS), which measures how closely their tracked regions overlap in generated and recorded frames. It is a visual-match measure, not a robot-task success rate. In the 30,000-hour model, which also received an outline of the person’s hand and upper-body pose as an input, hands scored about 0.785 and objects 0.527. The object-focused method raised the latter to 0.546; the authors report this as a 3.7% relative improvement.
During training, the researchers added more random noise to the model’s internal video representation for moving objects near the hands. That made those regions harder to reconstruct from appearance alone, so the model had to use surrounding context and motion to predict how the object would change. The authors found no drop on a separate perceptual image-quality measure.
Examples show the remaining errors: a model could place a hand correctly yet fail to lift a spoon, misrender a paper fold or make a cucumber disappear.
A separate evaluation fine-tuned Cosmos 3 Nano on demonstrations from 11 tasks in a simulated two-handed humanoid benchmark, then tested predicted frames on policy rollouts against simulator results. Compared with the released Cosmos 3 weights, the full setup, which combined 30,000 hours of human-video pretraining, a hand-and-body pose map and object-focused training, raised hand SCS from 0.63 to 0.88 and object SCS from 0.70 to 0.79. No physical robot was tested.
Training data and optimization steps increased together, so the study does not isolate the effect of data alone. The training and evaluation recordings also came from one collection protocol, and the authors describe the test set as limited in scale. The findings apply to these tested video models, not every world model or deployed robot.
Sources
- “What 30,000 Hours of Ego-centric Video Does Not Teach,” full paper on arXiv.
- The authors’ project page, with dataset, metric and model comparisons.
Figure 1 from the authors’ arXiv preprint, used under CC BY 4.0.