PointZero learned a reusable model of object motion before receiving any robot action labels. When researchers later trained it on 20 labeled demonstrations for each of three simulated manipulation tasks, the pretraining raised average success from 80.5% to 88.2%. The result suggests that predicting how visible points move can reduce the task-specific robot data needed to learn a new action.
Robot dynamics models commonly learn from a machine's recorded commands and the resulting scene changes. That makes each dataset tied to a particular arm and task, while ordinary videos cannot be used without reconstructing the robot's actions. The new pretraining objective replaces those commands with 3D point tracks: the paths followed by selected locations on a cloth, hinged object or rigid body over time.
PointZero starts with one color-and-depth image, turns its foreground pixels into a 3D point cloud and adds visual features that describe the scene. One to three known point tracks show how a small part of the object moves. A diffusion transformer then predicts ten-frame trajectories for every observed point. The team trained variants for about two days on eight H100 GPUs using 2.9 million synthetic frames of deformable, articulated and rigid objects. For a robot task, the same model can replace the sample tracks with the arm's end-effector poses or add a head that predicts action sequences.
The matched ablations isolate what that pretraining contributed. With downstream point-track supervision, a randomly initialized PointZero averaged 80.5% success across block stacking, microwave opening and glass righting; the pretrained version reached 88.2%. Without that auxiliary supervision, pretraining raised the average from 74.1% to 80.0%. Each comparison used 1,000 simulated rollouts per task. These are internal pretrained-versus-scratch tests of the same architecture, rather than a ranking against every robot-learning system.
A broader evaluation added 100 actionless expert videos to the 20 labeled demonstrations for each of three simulated and four physical tasks. PointZero recorded the highest or joint-highest success on six of the seven tasks against four application-specific baselines. Separately, models trained only in simulation were tested on 124 human interactions with 14 unseen real objects and led 11 of 12 reported motion-error metrics. Some generative results selected the lowest-error prediction from ten samples using the ground truth, so those numbers measure whether a good trajectory was generated, not whether a robot could identify it online.
The evidence makes 3D point-track completion a credible pretraining signal for object dynamics, especially where action-labeled robot data are scarce. It does not yet show broad physical competence: the synthetic data omit much of the clutter, contact forces, material variety and long-horizon motion found outside the lab, and the downstream tests remain small and task-specific. The paper and project page say that data, checkpoints and training code are released, but the linked repository currently contains only a teaser and says the release is coming soon. The next test is a reproducible multi-scene evaluation with disclosed physical trial counts and an online method for choosing among generated trajectories.