.jpg)
A generative model helped researchers turn human hand demonstrations into simulated data for robot hands. Using 780 motions from 210 first-person HOT3D video recordings, researchers at Meta Reality Labs Research and several universities report creating more than 220,000 feasible robot-hand trajectories across about 3,300 object geometries. Their Oct. 8 preprint evaluates the trajectories in simulation and reports no physical robot tests.
A person's fingers and a robot hand have different joints and contact behavior, so a captured human motion cannot simply be replayed on the machine. Inverse kinematics can quickly map a target hand pose to robot joint positions, but it does not ensure that the robot can maintain a grasp or move an object as intended. A physics-based optimizer can test those interactions in simulation, though it may need to search many candidate motions for each new demonstration.
The researchers call their approach Generative Neural Retargeting, or GNR. It starts with an inverse-kinematics path from the human hand motion and uses a flow-matching generator, which turns random starting points into candidate adjustments to robot controls, with the object's geometry as context. The team trained GNR on 28,000 demonstrations that the DIAL-MPC sampling optimizer had already solved. A physics simulator then checked candidate motions for object tracking, grasp force and other contact constraints.
On a separate test of 1,000 demonstrations involving unseen objects, grasps or geometry variants, GNR achieved a 56.2% success rate with 256 candidate trajectories per demonstration. The DIAL-MPC optimizer used to generate the training examples reached 27.2% with 3,000 samples. The authors report that GNR used 7.6 times less CPU time per trajectory in this comparison; both success rates were measured in simulation.
For the larger data set, the researchers augmented 780 human motions from 210 recordings into 360,150 simulated cases by varying object pose, geometry, physical properties and motion timing. DIAL-MPC solved 95,344 cases; GNR found feasible trajectories for 127,136 more that the optimizer had failed to solve. One appendix table reports a combined total of 222,480 trajectories, while a later summary gives 222,806; the paper rounds the corpus to about 223,000 across 3,300 geometries. The authors do not reconcile the small difference. Transferring a learned policy to a physical robot is future work.
Sources
Image credit: Oak Ridge National Laboratory via Wikimedia Commons, CC BY 2.0.