A new preprint describes how a 20–30-minute phone scan can help researchers adapt a robot policy—the learned controller that chooses its actions—to pick up objects in the captured room. In a small physical test, a Stretch 3 robot met the reported success criterion on 63.9% of 36 attempts across 12 tasks in one scanned office.
Called Splat2Act, the method combines 3D Gaussian Splatting, a way to render a scene from new viewpoints, with a mesh that supplies collision geometry. Using the reconstructed room and photos and dimensions of target objects, it finds reachable places to set objects down and plans pickup tasks.
A simulator-side “oracle”—a scripted planner with privileged access to exact object poses and scene geometry—uses the simulator’s robot state to generate successful pickup demonstrations. The trained policy does not receive those exact object poses or scene geometry; it takes head- and wrist-camera images, a readout of the robot’s own state and the task instruction, then returns a short sequence of commands for its arm, lift, wrist, base and gripper. Researchers use the demonstrations to fine-tune a pretrained vision-language-action (VLA) model in two stages: first with public-object examples from either the target scene or a wider mix of scenes, then with target objects on surfaces in the captured scene. Simulation and model training still require computing resources. The paper reports about 60 successful oracle demonstrations per A40 GPU-hour on average in its setup.
Across 101 scanned campus environments, the pipeline identified 852 reachable places to set objects down (receptacles) and generated 68,021 successful simulated pickup demonstrations. The authors evaluated policies in only two captured scenes: an office and a conference room. In the office, they tested 40 held-out robot-and-object setups that varied the robot’s starting pose and target placement; harder trials added distractor objects. The office route first adapted on public-object examples from that scene, then personalized on the target objects there. It reached 62.5% success, compared with 45.0% for direct fine-tuning from the pretrained policy.
On the physical Stretch 3, the office-trained policy succeeded on 63.9% of 36 attempts across 12 tasks on three office receptacles; its simulated counterpart succeeded on 66.7% of 12 attempts. The paper counted a trial as successful only when the robot lifted the target from its starting position and the policy sent a signal that it was done; a stop without a pickup counted as a failure. These are small, task-specific tests, not a broad estimate of real-world reliability.
Performance fell on two unseen lab tables: the two training setups scored between 22.2% and 33.3%, and which setup did better changed between tables. That result leaves generalization to other rooms unresolved. The work is an arXiv preprint; its authors say code and the dataset are coming soon.
Sources
- Xiaomeng Ye et al., “Splat2Act: From Captured Scenes to Personalized VLA Manipulation,” arXiv:2610.13945v1, submitted October 9, 2026.
- Authors’ Splat2Act project page.
- Illustrative photo, not from the study: ER-FLEX mobile manipulator by Auledas, used unchanged under CC BY 4.0.
