A new humanoid-robot benchmark finds that tool use becomes less reliable when a task adds movement. On one simulated ball-moving task, FastWAM, a system directing the robot’s actions, succeeded in 76% of attempts while standing and 9% when it had to walk to the target while carrying the stick.
HumanoidToolBench tests three jobs: push a ball beyond arm’s reach with a long stick, retrieve one with a hook, or break ice blocks with a hammer. At the first level, the robot must pick up the suitable tool. A stationary task adds using it; a mobile task places the target beside the tools before the trial, requiring the robot to walk over while carrying its tool.
The benchmark also tests tool choice. A standard set presents the suitable tool among unrelated objects. A decoy set adds a competing tool that lacks a property the job needs, such as a shorter stick for pushing the ball. The benchmark's staged levels distinguish tool pickup from task completion; decoys test whether a system chooses a tool with the property the job needs.
Researchers evaluated seven trained control systems in simulation, each on 100 episodes per task after one training run. In the standard ball-moving task, FastWAM succeeded in 76 of 100 stationary episodes and 9 of 100 mobile episodes. The paper also reports that success fell from stationary to mobile versions for every system that had nonzero stationary success.
Three systems were tested on a Unitree G1 humanoid, but only on the two stationary ball tasks, with 10 trials per condition. GR00T N1.7 completed 6 of 10 standard ball-moving trials and 4 of 10 standard retrieval trials. The hardware evaluation did not include walking while using a tool.
The team released HumanoidToolBench and ToolBook, which contains 3,003 demonstrations collected by teleoperators in simulation and 91 demonstrations collected on a real G1. The paper is an arXiv preprint, and each system was trained once. These results show how a benchmark can expose failures between tool selection and task completion; they do not establish how humanoids will perform in workplaces or on other tasks.
Illustrative photo: RuinDig/Yuki Uchida, Wikimedia Commons, CC BY 4.0. The robot is shown at the Japan Mobility Show, not in the reported experiments. Photo source.
