A paper posted to arXiv on Oct. 15 introduces P³-WAM, one model that can reconstruct hand and camera movement from first-person video, predict video from a prescribed movement, or plan a manipulation toward a target image. The authors also show two executions on a physical robot after converting its predicted human-hand motions into robot commands; those demonstrations run without feedback during the task.
One model links video and movement
P³-WAM works with video and structured 3D actions: the movement of the hands and camera. Its Perceive mode estimates those actions from a video. Predict generates future video when the actions are supplied. Plan takes an initial image, an instruction and an optional goal image, then generates both a video and the hand-and-camera trajectories for the task. A goal image can specify where an object should end up when words alone leave that arrangement unclear.
The paper adapts the pretrained Wan2.2-Fun-A14B-Control video model so its video and action streams exchange information. During training, the model sometimes receives video and predicts motion, and sometimes uses motion to predict video. The authors say the shared training improved all three modes compared with training each mode on its own across four datasets.
In matched tests across those datasets, unified training lowered average camera-space hand-joint error from 113.5 to 86.9 millimeters. It also raised PSNR, a pixel-level similarity measure, from 17.36 to 18.14 dB and lowered FVD, a video-feature distance, from 1,105.95 to 1,009.84. These benchmark metrics do not measure robot task completion.
Planning videos scored higher on four measures
For planning, a Qwen3.8-Max vision-language model that did not know which method produced each video rated 188 videos across 17 task categories on a five-point scale. The researchers compared P³-WAM with Wan2.2 I2V-A14B and Cosmos 3-Nano, each tested with and without a goal image. P³-WAM scored higher than the best baseline setting on four of five measures, including instruction adherence (3.739 versus 3.119) and goal transition (3.841 versus 3.178). Its score for temporal and object-identity consistency was slightly lower than the best baseline: 4.651 versus 4.684. These are video ratings, not robot success rates.
Two robot demonstrations still ran open loop
The researchers also tested whether the generated motions could be executed on a physical robot. They reconstructed objects and motion from the generated video, converted the human-hand path into robot commands and replayed it in two experiments. The authors describe two demonstrations, while the project page shows fruit transfer and block manipulation. The robot does not execute the generated video itself, and it receives no new observations to correct its path once execution begins.
Because execution is open loop, these demonstrations do not show whether the robot can recover from a slip or an unexpected object position. The authors say absolute hand position is ambiguous from a single camera view and identify depth or touch sensing and closed-loop feedback as future work. They report training on 16 H100 GPUs for about 450 hours; the project page says code is forthcoming. The work is an arXiv preprint, and the reported scores come from the authors' benchmarks and a model-based judge.
Sources
P³-WAM paper and appendix; authors' project page and supplementary video. Illustrative lead photo, unrelated to the study: Photo by lost places via Wikimedia Commons, CC BY 2.0.
.jpg){kind=link}