Researchers led by Amazon’s Fundamental AI Research group used four human videos to train a Unitree G1 to pick up, carry and drop unfamiliar objects. The system generated new interactions from those clips, trained one robot policy in simulation and then tested it on the real humanoid without real-world policy fine-tuning.
The team started with videos of people carrying boxes. A video-generation model produced 256 alternative clips with boxes, balls, bins and barrels, changing object size, appearance and position while keeping the scene and camera view. Some clips also showed the person adapting how they handled each object. The researchers call these counterfactual examples: plausible interactions that did not happen in the original recordings, rather than additional real-world demonstrations.
Generated video alone is not a robot trajectory. The video clips provide only one camera view, leaving the scene’s 3D shape and motion uncertain; a plausible-looking interaction may still be physically impossible. The PRISM pipeline uses contact between a person’s hands and an object to stabilize the reconstructed motion, then adjusts that motion to fit the humanoid before training in simulation. Of the 256 generated clips, 137 produced feasible robot-object trajectories; severe collisions were the main reason the rest failed the solver.
In real-world tests, the policy succeeded in 55 of 60 trials on new objects from the four categories shown in generated videos, and 32 of 40 trials on eight object types absent from those videos. Each object was tested five times. A trial counted only if the robot grasped the object and carried it at least three meters without dropping it or falling. A human operator still supplied joystick commands, while an external computer estimated depth from the robot’s stereo camera.
The test covers one commanded pick-carry-drop skill, not a robot choosing its own tasks. The scale is also small: four seed videos yielded 137 feasible trajectories, which the authors say is too little to establish how performance changes as training data grows. Their simulator treats objects as rigid bodies and does not capture each item’s exact mass distribution or surface properties, leaving articulated and deformable objects outside the demonstrated result.
Sources
- Zihan Wang et al., “Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation” (arXiv, September 29, 2026)
- Authors’ PRISM project page and demonstrations
- Representative image, not from the experiment: “Japan Mobility Show 2025,” by RuinDig/Yuki Uchida, CC BY 4.0; resized by Wikimedia Commons.
