A simulated robot can waste training on setups it has already mastered or cannot yet solve. University of Washington researchers report a way to redirect that practice: Success-Guided Sampling (SGS) favors task setups where the current controller sometimes succeeds. The Oct. 8 paper reports improved results on difficult simulated walking and assembly tasks.
In reinforcement learning, a controller learns by trying actions and receiving feedback. Researchers can run many copies of a robot simulator in parallel, but uniform sampling gives every setup the same chance, even when a policy already handles it or is still far from solving it. That can spend compute on less informative experience.
SGS keeps the last 100 success-or-failure results for each setup in a fixed set. A setup pairs a task with a starting state and goal; for an arm, it can specify whether a part is already grasped or must be picked up before assembly. The sampler selects setups where success is neither very rare nor nearly certain more often, while still revisiting every setup. It adjusts practice without a hand-written easy-to-hard order. The authors say the algorithm is not a fundamentally new curriculum principle; they test a simple success-based sampler with diverse resets and large-scale training.
At 1,048,576 parallel simulated environments, a quadruped policy averaged 73% success across 13 terrain types, compared with 54% using Prioritized Level Replay, a competing sampler. On Franka nut-and-bolt assembly, SGS reached 70%, versus 6% with uniform sampling and 5% with Prioritized Level Replay. These results average three training seeds. The million environments are virtual simulation runs, not physical robots or hardware trials.
The paper also reports a separate transfer test on a physical UR5e arm. The team trained policies to read two color-camera views and the arm’s measured state by imitating controllers learned in simulation, then ran them without real-world fine-tuning on three assembly tasks. In the physical trials, the arm completed 18 of 48 nut-and-bolt trials, 30 of 49 rod insertions and 47 of 50 gear assemblies. The corresponding camera-based policies scored 89%, 93% and 96% in simulation, so the transfer gap varied substantially by task.
SGS still depends on a fixed, discrete set of task setups and on a separate method for creating diverse reset conditions; it does not solve how to cover continuous or much larger task spaces. The work is an arXiv paper, and its hardware results come from a small, controlled set of trials rather than routine deployment.
Sources
- The paper, including methods, simulation comparisons, hardware trials and limitations
- Authors’ project page and demonstrations
- Lead image: Figure 1 from Octi Zhang and colleagues, “A Balanced Data Diet,” licensed CC BY 4.0; unmodified paper illustration.