Imagine3D-LLM, a 7-billion-parameter vision-language model, scored higher on six 3D-scene benchmarks after researchers trained it to reconstruct a compact scene from multiple images while learning to answer spatial questions. Against a baseline with the same model, data and schedule, its SPAR-Bench score rose from 60.9 to 68.5.
Figure: Imagine3D-LLM architecture. Source: the authors’ paper, shared under CC BY 4.0.
Models can identify objects in separate views yet struggle to connect their positions. Researchers from KAIST, ETH Zurich and Google added learnable “Gaussian summary tokens” to the model’s visual sequence. At an intermediate layer, those summaries are decoded into 3D Gaussian Splatting, a compact arrangement of 3D blobs that can be rendered back into the input views. The summary tokens sit before the question tokens, so the language model can attend to the scene representation when generating its answer. Training rewards both a faithful reconstruction and a correct language answer.
The matched baseline kept the LLaVA-Video-7B backbone, training data and schedule but omitted the summary tokens and reconstruction/distillation objectives. The full method improved all six reported benchmark metrics, including SQA3D from 56.5 to 63.8. The paper also reports 68.5 on SPAR-Bench versus 63.3 for 3DThinker-7B, but that is a cross-paper comparison rather than a controlled test.
The experiments use indoor scenes and one 7B backbone. The authors flag training efficiency and the fixed scene-representation budget as limits, especially for larger outdoor environments; the project page lists code as coming soon. The result makes a case for scene reconstruction as a training signal for multi-view reasoning; transfer to other backbones and outdoor scenes remains untested.