Researchers at KAIST have introduced Tetris3D, a model that reconstructs a 3D scene from one image by generating objects in relation to their neighbors. The approach uses cues such as support, contact and containment to make objects fit together, including where their meeting points are hidden from view.
When a scene generator builds each object separately, the finished pieces can overlap or leave gaps. Tetris3D instead infers which objects support or contain others, then generates them in that order. A table, for example, is reconstructed before a cup that rests on it.
For each new object, the model uses the surrounding surfaces and their physical relationship as spatial guidance. For parts hidden by occlusion, the model uses visible regions to guide completion, while neighboring geometry helps infer the covered contact areas.
The team trained the model using ComOb, a simulation-built dataset of 1.2 million scenes and 3.4 million annotated examples assembled from about 400,000 object models. The scenes include arrangements such as stacking, leaning and containment, with object meshes and relation labels.
In a Toys4K evaluation, researchers placed objects into simulated interaction scenes and supplied the methods with ground-truth depth maps, so their depth values came from the reference scenes rather than image-based estimates. Tetris3D recorded a penetration-depth score of 0.0036, compared with 0.1177 for the best listed baseline on that measure; the score measures how far generated objects intrude into one another, so lower means less overlap. In the real-scene Picasso benchmark, the pipeline instead used depth predicted from the image and relationships inferred by a vision-language model. Tetris3D scored 0.0016 on penetration depth, compared with 0.0111 for the best listed baseline.
The physical cues are learned guidance rather than hard constraints, so the system cannot guarantee that every reconstruction will be physically valid. It also relies on separate models to estimate depth, segment objects and infer their relationships; mistakes in those inputs can carry forward as the model generates later objects. The paper is a preprint, and it does not report a robot manipulation deployment.
Image: Figure 1 from the authors’ preprint, used under CC BY 4.0.