A new robot-learning study found that labels describing the quality of human demonstrations improved simulated-task averages for three tested AI policies, but the size of the gain and the best way to use those labels varied by model. In one physical-robot test, adding quality conditioning raised success from 31.7% to 32.5% after simulation pretraining.
Researchers from Georgia Tech, Axis Robotics and universities in the United States and Singapore built AXIS-Bench to test how robot systems learn from demonstrations that include pauses, detours and other imperfect movements. A vision-language-action (VLA) policy turns camera images and task instructions into robot actions. Researchers often first train these policies with behavioral cloning, in which a model learns to copy human demonstrations, then fine-tune it on specific tasks.
AXIS-Bench records object positions and contact events in simulated demonstrations, letting researchers estimate whether each movement made progress toward a task. They compared three ways to use that estimate: weighting gives more influence to higher-quality examples during training; filtering removes lower-scoring examples; and conditioning adds a quality label to the task instruction so the policy can learn from it. Labels could describe a whole demonstration or a short action segment.
Quality labels helped in simulation, with model-specific trade-offs
The simulation comparison used 5,000 human demonstrations, 24 tabletop tasks and three policies: π0.5, SmolVLA and X-VLA. Within each policy, the researchers matched the pretraining data and training budget. They then tested each configuration across the same 50 starting layouts for every task, for 1,200 trials per configuration.
When the quality score separately accounted for the robot’s approach to an object and its later manipulation, labeling the whole demonstration improved mean success over plain behavioral cloning for all three policies. The reported relative gains were about 1% for π0.5, 74% for SmolVLA and 8% for X-VLA. Those percentages use a different baseline for each policy; SmolVLA’s plain-cloning success was 5.08%, compared with 57.42% for π0.5 and 25.75% for X-VLA.
The results did not identify one best recipe. Conditioning tended to work better with whole-demonstration labels, while filtering often favored labels for shorter action chunks, but the preferences changed with the model and the quality measure. In a three-seed replication, conditioning beat plain cloning in six of nine model-and-seed comparisons, tied once and scored lower twice.
Only a small extra gain appeared on a physical robot
The team also evaluated one π0.5-based policy on a Franka FR3 robot in a single tabletop workspace. It covered ten tasks, with 24 trials per task. Fine-tuning without simulation pretraining reached 9.6% success; plain behavior-cloning pretraining on simulated demonstrations raised that to 31.7%. Quality-conditioned pretraining reached 32.5%, an additional 0.83 percentage points over plain pretraining.
That physical test is a limited reference experiment, not evidence of a general deployment gain: it used one policy and workspace, and the conditioned setup also used quality tags during later task-specific training and when choosing actions. The paper is an arXiv preprint submitted on Oct. 13. Its authors’ project page says the benchmark’s code and dataset are coming soon, so other researchers cannot yet use those linked materials to reproduce the comparisons.
Sources
- Full paper: What Matters in Quality-Aware VLA Pretraining
- arXiv record and abstract
- AXIS-Bench project page
Image: AXIS-Bench teaser diagram by the authors, from the project page; CC BY-SA 4.0.