Five robot-control models completed fewer than half their trials in a new simulation test that changes how objects behave, not just how they look. Across 50 existing two-arm tasks retested with varying physical conditions, their average success rates ranged from about 32% to 44%, according to a September 22 technical report introducing RoboTwin-Phys.
Changing weight, friction and other physical properties
Robot benchmarks often vary an object's position, a scene's layout or the camera view while keeping properties such as weight and friction fixed. A robot policy can therefore score well on visually different scenes without facing a heavier object or a more slippery surface. RoboTwin-Phys extends the existing RoboTwin-2.0 simulation with 13 changing attributes, including object mass, friction, center of mass and arm-joint damping.
The researchers sampled a new combination of conditions for each simulated episode and kept it fixed during that attempt. They also ran an expert planner beforehand and excluded combinations it could not complete, so the test would not count impossible tasks as model failures. The report describes more than 5,000 expert demonstrations annotated with the physical settings used in each episode; those annotations could help developers trace which settings accompany particular failures.
Results from 50 simulated tasks
Under the new physical-variation setting, success averaged 31.60% for the lowest-scoring model and 44.24% for the highest. Each of the 50 tasks had 100 rollout episodes per model. Some jobs, such as handling varied bottles, were difficult across models, while others held up better. This points to a measurable weakness in these policies under the benchmark's combined physical and visual variation.
The report also lists much higher scores for the same models under the older visual-and-layout setting. Those reference scores come from separate public benchmark evaluations, however, whereas the physical-variation scores were measured by the new study's authors. The difference is a useful warning signal, not a controlled percentage-point estimate of what changing weight or friction alone caused. The work tests simulated tasks; it does not measure performance on physical robots.
.jpg){kind=link}