Researchers have developed a way to turn a photorealistic room scan into a robot simulator where selected objects can move and make contact. Their PhiRIE system improves the geometry of reconstructed objects in tests on 50 indoor scenes, but a separate manipulation test shows how much work remains: a robot completed 128 of 480 planned tasks with the converted objects, compared with 385 in the original simulation environment.
A room reconstructed with 3D Gaussian splatting can look convincing from many viewpoints. The image representation, however, does not tell a simulator which points belong to a cup, what shape the unseen underside has, or what should appear on the table after the cup moves. PhiRIE links those decisions: it selects an object, builds a movable replacement, removes the object's old appearance from the scan and fills the newly exposed background. A physics engine handles contact and motion; the reconstructed appearance follows the simulated object's position.
Better geometry means slower builds or fewer objects
For 1,871 object requests across 50 ScanNet++ scenes, the researchers compared several ways to choose and align generated objects. With the same combined pool from two object generators and 1,800 objects retained, choosing candidates from observed geometry raised F1 at a 20-millimetre tolerance from 0.336 to 0.348. Trying alternative alignments raised it to 0.383. F1 here measures how closely reconstructed object surfaces match independently evaluated reference geometry; it is not a robot task-success rate. The extra alignment attempts also roughly doubled construction time, from 35.68 to 70.82 minutes per scene. Those times exclude scene capture and initial Gaussian reconstruction.
The paper's stricter acceptance setting achieved higher geometric F1, 0.425, but kept only 399 of 1,871 requested objects. Of those 399, 134 had independent matches; 17 of the 134 showed collapsed geometry. This is a trade-off between usable object coverage and geometric quality, rather than a general accuracy gain for every object in a room.
Task results remain well below the original simulator
In a separate RoboCasa manipulation test, the team held the robot policy and task settings fixed across 480 planned trials. PhiRIE's full constructor produced 128 successes, versus 73 when objects came from one generator alone. But a simpler method that consistently preferred one generator over the other produced 131 successes, while the original simulator environment produced 385. The paper says its geometric improvements from candidate selection and retry do not establish an additional manipulation gain over the closer two-generator comparison.
The test replaced target objects while leaving the room and destination native to the simulator. It used idealized posed RGB-D observations to reconstruct those targets, and the robot policy saw native mesh renderings rather than PhiRIE's Gaussian visuals. The result therefore measures a controlled asset-replacement task, not performance of a robot trained in a fully scanned room or transferred to a physical site. The method currently assumes rigid objects, assigned physical properties and locally planar background filling.
Sources
- PhiRIE research paper, arXiv:2609.26795, submitted September 22, 2026
- Full paper with methods and Tables II and III
- PhiRIE project demonstrations
- Lead image: Figure 1 from the PhiRIE paper by Runyi Yang and colleagues, CC BY 4.0.