A general-purpose vision-language model completed towel and notebook pickup after receiving one human demonstration video, while the same agent failed every run without the video. Each condition contained only three real-robot trials, but the result shows that a visual example can change physical behavior without retraining the model or supplying robot action labels.
The researchers built GPT-Policy to test in-context robot learning: adapting from information supplied during a task rather than updating model weights. A dedicated robot policy normally learns a fixed mapping from observations to actions during training. GPT-Policy instead gives a fixed model the current camera views and robot state together with a task instruction and optional context, such as a human video, a robot demonstration, a target image or earlier interaction history.
That context passes through a constrained control loop. The vision-language model proposes one structured tool request, such as a target gripper pose. A robot adapter checks inverse kinematics, joint limits, timing and fresh feedback before execution. Accepted motions produce new observations; rejected or incomplete motions return errors and images for the next decision. This separates deliberative task reasoning from the checks needed to turn its output into arm movements, although the paper says the Cartesian planner does not check collisions.
Tests used GPT-6 Astra on ten tasks, with three trials for each reported condition. Human video raised both towel and notebook pickup from 0 of 3 successes to 2 of 3, while also reducing average decisions and run time. Contact-rich tasks exposed the value of numerical action context. Bottle-cap removal rose from 0 of 3 with no demonstration to 2 of 3 with video and 3 of 3 when recorded poses, gripper states and commands accompanied the video. Plug removal and reinsertion stayed at 0 of 3 with video alone and reached 2 of 3 with the aligned actions. Six target-image, history and human-interaction tasks reached 3 of 3, but those conditions lacked no-context baselines.
The evidence supports using a general model as a slow task-level adapter when a new demonstration is available; it does not establish a safe replacement for a trained low-level controller. Runs often took 8 to 26 minutes, and the researchers repeatedly observed collisions between the two arms. The public repository includes hardware adapters, but its license remains pending and the simulation evaluation pipeline is not yet released. The next test should report larger matched trial sets, collision and intervention counts, independent outcome checks and comparisons with dedicated robot policies on the same unfamiliar tasks.