An Oct. 1 preprint from Technical University of Munich researchers introduces a robot-learning method that trains movement policies from scores assigned to sampled actions. The method avoids differentiating a learned value model with respect to each action; its evidence so far comes from simulation.
In continuous-control learning, a policy is the rule a robot uses to choose actions. A learned critic estimates how much future reward an action may earn. Some algorithms use the critic's estimate of how changing an action would change its predicted return to improve the policy. But a critic trained to estimate return values may not provide a reliable estimate of that change. The new method, Forward Entropy-Regularized Policy Optimization (FERPO), is designed to avoid that step.
FERPO draws candidate actions from the current policy and scores them with the critic. Each action's score, adjusted for how likely it was under the current policy, determines its weight in a target distribution. FERPO then trains the policy to reproduce that weighted distribution of actions. A built-in rule monitors effective sample size, which shows whether the weights are spread across enough actions, and limits how much a few samples can dominate. The method still uses gradients to train the policy and critic; it is not gradient-free.
The researchers compared FERPO with Relative Entropy Pathwise Policy Optimization (REPPO), using the same actor and critic architectures, across 22 DeepMind Control Suite tasks, four G1/T1 locomotion tasks and eight ManiSkill manipulation tasks. After 50 million environment interactions, FERPO's interquartile-mean return on DeepMind Control Suite was 897.7, compared with 891.0 for REPPO. The adjusted confidence interval for the difference included zero, so the result does not establish a clear advantage. ManiSkill success-rate scores were also close: 95.67% for FERPO and 95.52% for REPPO, with an adjusted interval that likewise included zero.
On the four simulated G1/T1 tasks, REPPO had the higher aggregate return: 34.63 versus 33.22. REPPO led on both G1 tasks; FERPO's estimate was slightly higher on flat T1 terrain and similar on rough T1 terrain. The paper therefore reports competitive but mixed results, not an overall win.
A separate timing test on one A100 used fixed synthetic rollouts and 16 candidate actions per state. FERPO reduced combined actor-and-critic update time by 6.9% without caching and by 22.2% with cached values. That measures update time, not full simulation or end-to-end robot-training speed; the cached setup also used 25.6% more peak GPU memory on CheetahRun.
FERPO offers robotics researchers another way to update continuous-control policies when they want to avoid relying on critic action derivatives. The preprint reports no physical-robot test, and it identifies critic-value error, finite-sample error and higher memory use as limitations. An on-robot evaluation and memory measurements during full training remain important next checks.
Sources
- FERPO preprint, v1 (Oct. 1, 2026); authors' implementation.
- Illustrative image: MuJoCo Menagerie, Unitree G1 rendering. Unitree Robotics copyright and BSD-3-Clause terms are listed in the model license. The image is not from the FERPO paper.