Robot world models generate video of what a robot might see after taking an action. But a sequence can look plausible frame by frame while drifting into an inconsistent 3D scene. DepthWorld adds camera-depth data to training, and its authors report better image predictions from external cameras.
A robotic arm assembles a component in a laboratory. Photo: Shixart1985 / Wikimedia Commons, CC BY 2.0.
The team built DROID-3D by adding metric depth and updated camera calibration data to the DROID manipulation dataset. The authors’ project page says it covers 71,100 episodes from 28 robots across 13 labs. Depth maps estimate how far visible surfaces are from each camera. DepthWorld uses a Stable Video Diffusion model to predict several camera views and their depth maps together.
The paper’s results section reports that DepthWorld improved peak signal-to-noise ratio (PSNR), a measure of image similarity, by 1.48 dB for external views and 1.02 dB for the robot’s wrist view, compared with an RGB-only model trained with the same budget. But the displayed table scores imply gains of 1.46 dB (24.09 versus 22.63) and 0.98 dB (17.96 versus 16.98). The paper does not explain these discrepancies.
The evaluation uses 256 held-out episodes from robots and labs also represented in training, so it does not test transfer to new robots or labs. It evaluates generated rollouts rather than physical task success. The authors also report that prediction errors can compound over autoregressive rollouts.
Sources
DepthWorld paper (arXiv:2610.08780v1); authors’ project page; image source.
