Researchers have built a vision model that estimates how each pixel moves and turns in 3D from video recorded by one camera. The model, MoSE3, also groups pixels that move as parts of the same object. The paper also evaluates nine real-world clips of a robot arm manipulating rigid objects, scoring motion estimates rather than task success.
A 3D point tracker follows where selected image points go over time. Those paths show translation, but not which points belong to the same moving part or how that part rotates. A robot opening a hinged door, for example, needs the door's rotation as well as the path of points on its surface.
MoSE3 predicts 3D point tracks and a learned code that groups pixels with similar motion. A differentiable geometric fit combines them to estimate a rigid transform: translation and rotation in a shared world coordinate frame. The model makes this prediction in one forward pass from ordinary RGB video, without a depth sensor or per-video optimization.
The researchers also introduce Art-Kubric, a synthetic dataset of articulated objects moving and interacting in physics-based scenes. It supplies dense 3D tracks, each articulated link's pose and labels for which pixels share rigid motion, giving the model examples of parts moving together or separately.
On iTACO, an articulated-object benchmark, the authors report an intersection-over-union score of 0.784 for grouping moving parts, compared with 0.353 for Track4World-Pi3x in the same table. The score measures overlap between predicted and labeled part regions. MoSE3 depends on another model, pi-cubed, for camera and scene-geometry estimates, so errors there affect its motion output; the authors also report lower accuracy under fast motion. Deformable-object results are qualitative.
Sources
- MoSE3: Learning World-Space SE(3) at Every Pixel, arXiv:2610.03716v1
- Authors' project page, visualizations and benchmark summary
- Illustration: MoSE3 paper teaser by Jiahuan Cheng, Zhiyi Li, Tian Xia, Ruojin Cai, Yilun Du and Qianqian Wang, used unmodified under CC BY 4.0.