A robot can complete a chore and still ignore how someone wants it done. A new benchmark gives researchers a way to measure both outcomes: whether a robot finishes a manipulation task and whether it follows a person’s instructions about how to carry it out.
The project, called LIBERO-PeRM, was submitted to arXiv on Oct. 15. Its project site lists it in NeurIPS 2026’s Evaluations and Datasets Track. The researchers built it to address a gap in robot testing: a system might place a mug on a table, for example, without turning the handle outward as requested or choosing a coaster to protect the surface.
The benchmark scores the task and the preference separately
LIBERO-PeRM uses a simulated manipulation environment built on the LIBERO robotics benchmark. The policy receives camera images and a language prompt describing both the task and the preference, then outputs robot actions. The benchmark groups preferences into three types: motion, such as avoiding an area; skill, such as orienting a mug handle outward; and user-profile choices, such as selecting a coaster.
The five suites include 312 tasks overall, including tasks without a preference for comparison. Separately, the authors defined 37 single preferences and tested 18 two-preference combinations. The authors report 40,800 rollout episodes. They evaluated six vision-language-action policies, which turn camera input and written instructions into control actions.
The separate scores reveal a gap a simple success rate can hide. In Table 2’s zero-shot test on LIBERO-SinglePref, the π0.5 policy reached the task goal on 66.67% of all-preference trials, while its separate preference-satisfaction score was 37.92%. That second figure is not the share of trials that fully complied: it averages a four-level score from 0 to 100, with failed task attempts counted as zero.
Fine-tuning helps, but preferences still fail to transfer cleanly
Across the tested policies, the authors found that following preferences without preference-specific training was difficult, while supervised fine-tuning made the preferences more learnable in the benchmark. Mixing preference and ordinary task examples helped preserve performance when no preference was given, but weakened learning of motion preferences compared with preference-only training. Generalizing a preference to a new layout or task, combining preferences, and handling conflicts remained difficult.
These results describe simulated policies, not robots in homes. The benchmark covers predefined preferences that can be expressed in language and scored automatically; it does not infer a person’s habits from past interactions or measure open-ended subjective preferences. The paper reports no real-robot or long-term user study, so the scores show what current systems do in this controlled test, not how reliably a household robot would adapt to an individual.
Sources
- LIBERO-PeRM paper on arXiv
- LIBERO-PeRM project page and benchmark materials
- Illustrative photo by Nenad Stojković (Shixart1985), via Wikimedia Commons, licensed under CC BY 2.0; the image is not from the study.
