Researchers at UC Berkeley and Amazon FAR report a way to train a humanoid motion policy from scratch in simulation and run it on a Unitree G1 without real-world fine-tuning. The method, QF3 (Fast Flow RL with Filtered Q-Gradients), tracked a roughly three-minute dance sequence on the robot. In a separate simulation comparison, it reached full-length motion-tracking episodes about ten times faster than the on-policy method FPO++.
QF3 is an off-policy reinforcement-learning algorithm: it learns from a replay buffer of actions already tried, rather than needing a fresh rollout for every update. Its flow policy turns noise into a multi-joint action through a sequence of small updates. That makes it expressive, but training can be unstable when a learned critic, which estimates future reward, steers the policy toward actions far from anything in its replay data.
The update combines flow matching with the critic’s action gradient. QF3 predicts the policy’s output using one update step in its flow policy, then clips the critic-driven correction in each action dimension around the replay action. If a correction crosses that limit, its critic gradient is masked; the flow-matching loss still pulls the policy toward the stored action. This keeps the critic’s influence near examples it has seen and avoids backpropagating through the policy’s full sampling process.
In the authors’ GPU-parallel Holosoma simulator, the policy controlled 29 joint targets on a Unitree G1. Across three random seeds, QF3 reached 500-step episodes for the LAFAN “dance1_subject2” motion in about half an hour; FPO++ took roughly ten times longer. The comparison used one GPU per run and the same GPU type for both methods on each task. For velocity tracking, QF3’s final reward was within about 4% of FastTD3’s after the same 10-hour budget.
The hardware result was a transfer test, not the speed comparison: the team says its simulation-trained G1 policy followed velocity commands and tracked the dance clip on the physical robot with no real-world fine-tuning, using the same joint-target action space and controller gains. The paper does not establish transfer across different humanoids, unfamiliar motions or real-world manipulation. It also reports separate simulated tests in which QF3 fine-tuned a pretrained bimanual policy; for putting bottles in a bin, success rose from 91% to 95% over 50 evaluation episodes, but this was not a physical-robot trial.
The work is an arXiv preprint submitted Oct. 6, 2026. The authors’ project page says the code is not yet available.
Sources
- Chung Min Kim et al., “QF3: Fast Flow RL with Filtered Q-Gradients”, arXiv:2610.08789v1, submitted Oct. 6, 2026.
- Authors’ QF3 project page.
- Illustrative photo: Unitree G1 at Japan Mobility Show 2025, by RuinDig/Yuki Uchida, CC BY 4.0. The image shows a trade-show display, not the robot in the QF3 experiments.
