A robot-learning system trained from one human video per task completed five manipulation tasks on a real, multi-fingered hand. For each method and task, the researchers ran 20 trials with starting and goal positions not shown in the videos; Dex-One2Many succeeded in 60% to 85% of these real-world tests, compared with 15% or less for two single-video baselines.
The method, called Dex-One2Many, uses each video to identify what must happen rather than to copy the person’s exact movements. A vision-language model turns the demonstration into a sequence of scene graphs: descriptions of relationships such as a hand grasping a tool, the tool contacting an object, and that object reaching its destination.
Those relationships guide reinforcement learning in simulation. The system generates many valid combinations of object positions and finger grasps, then trains a policy to complete each stage. Because the policy learns the task’s order and goals rather than a fixed trajectory, it can choose a different grasp or path when an object starts in a new position.
The team tested five tasks: moving a doll, can, or stamp, using a hammer, and sweeping with a tool. On the physical setup, a six-degree-of-freedom UR3 arm carried a 20-degree-of-freedom Wuji 1 hand. For each method and task, the researchers ran 30 real trials: 10 in configurations shown in the video and 20 with different starting and goal positions. The reported 60%–85% success range refers to those unseen configurations.
The result is promising but still a small laboratory evaluation, not evidence that a robot can generalize to arbitrary household work. The paper also reports simulations with four different arm-and-hand designs, but those used separately trained policies; the physical tests used one robot configuration. The authors note that inaccurate object-pose estimates and differences between simulated and real objects can still cause failures, and their reconstruction pipeline handles rigid objects rather than articulated ones.
Lee and colleagues posted the study as an arXiv preprint on Oct. 8. Read the paper.
Sources
Image credit: Figure from Dex-One2Many by Lee et al. (CC BY 4.0).