A new system called 4D-HOF turns a single-camera video of someone handling an object into a moving 3D reconstruction of the hand and object. The model, from researchers at NVIDIA and Texas A&M University, is designed to preserve contact and motion that can be hard to recover when a hand hides part of an object. In the paper, “4D” means 3D geometry tracked over time.
The reconstruction starts with rough estimates of hand pose, object shape and position from pretrained vision models. A conditional flow-matching model then refines the hand and object together: it learns a path that gradually moves an imperfect estimate toward a plausible interaction. During refinement, image silhouettes, hand-object contact and penetration constraints, and temporal smoothness guide the result.
On 18 selected HO3D-v3 sequences, the researchers report 97.27% contact accuracy versus 90.99% for CHOIR; the score is the share of frames whose contact state matched ground truth. On 20 selected DexYCB sequences, the paper reports mean hand-joint position error of 6.84 mm, compared with 19.06 mm for AGILE; lower is better. The team trained on HO-Cap, OakInk and SHOWMe, excluded both evaluation benchmarks from training, and used the same model weights on both.
The paper’s computational-cost table lists about 24 minutes total for 4D-HOF: roughly 23 minutes of preprocessing and one minute of refinement. It lists 30 minutes for CHOIR and 2.1–2.2 hours for AGILE. The table does not specify inference hardware, so those times are reported results rather than a hardware-normalized speed test.
The method still depends on the initial vision-model estimates; the authors report failures when those estimates are badly wrong. The current system handles one hand and rigid objects. The benchmark evaluates 3D reconstruction, not whether a robot trained from the reconstructed video can complete a task.
Illustration: Shiqi Li et al., “4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction,” arXiv:2610.08782v1, CC BY 4.0.
Sources
4D-HOF paper, arXiv:2610.08782v1; authors’ project page and reconstruction viewer.