Researchers have adapted a pretrained video generator to estimate a scene’s depth, camera movement and moving 3D points from a video clip. Their Gen4P model encodes the clip once, then a smaller decoder answers separate queries about selected locations and times.
Image: Yitong Xia, Zeren Jiang and Chuanxia Zheng / Gen4P project page, CC BY-SA 4.0.
The paper calls this “4D perception”: describing a three-dimensional scene and how it changes over time. From one video, Gen4P can return depth maps, camera poses and tracks showing where points move in 3D.
Gen4P reuses a video generator as a scene encoder
At inference, Gen4P passes the video through the encoder of a pretrained Wan2.1 video generator once. A smaller decoder then answers queries about selected points and times using the encoded clip. The researchers froze most of the original generator and fine-tuned a low-rank adapter and the query decoder to predict motion and geometry.
The one-pass description applies to encoding the clip, not to training or the full query process. The model was trained on about 200,000 clips from 11 public datasets. The paper reports 75,000 optimization steps on eight NVIDIA H200 GPUs.
The paper reports its strongest depth comparison on Sintel
The team evaluated 3D point tracking, video-depth estimation and camera-pose estimation. For depth, they tested on Bonn, Sintel and KITTI, datasets not used for training. The paper reports its strongest comparison on Sintel and competitive results on Bonn and KITTI.
These results come from an arXiv preprint. The paper does not establish performance in a deployed product or reliable reconstruction for arbitrary footage.
Sources
Gen4P paper, methods and benchmark tables
Gen4P project page and pipeline figure
CC BY-SA 4.0 license for the project-page material