A team of Zhejiang University researchers has introduced SpaceCast-Bench, a benchmark that tests vision-language models on spatial relationships in indoor scenes. Gemini 3.5 Flash scored 58.0% on the benchmark’s unweighted average across 16 task types; a human baseline from 12 volunteers reached 87.2%. The work is an arXiv preprint submitted Oct. 8.
Figure: Hongxing Li et al., “SpaceCast-Bench,” arXiv:2610.12402v1, CC BY 4.0.
The authors contrast this test with benchmarks that ask a model to identify spatial relations already visible in an image. SpaceCast-Bench also asks what happens after an object is moved or removed, or the full layout is rotated. One example asks a model to predict whether a cup resting on a book will move when the book is slid from beneath an envelope. The resulting scene is not shown.
The test contains 3,862 questions with answer choices, drawn from validation scenes in ScanNet and ScanNet++. Its 16 task types include static perception, local changes such as moving or removing an object, and rotations of the whole layout. Researchers evaluated 21 models. Because the overall score is the unweighted average of accuracy across task types, a type with fewer questions counts as much as one with more.
Gemini 3.5 Flash led at 58.0%; the 12 volunteers scored 87.2% on the same task set, while the paper’s random baseline was 25.9%. Models trained specifically for spatial tasks scored 25.7% to 29.1%, close to that baseline. The comparison measures answers to the benchmark’s questions, not whether a model can safely carry out a plan.
The authors also fine-tuned Qwen3-VL-4B on generated examples from separate scenes. Its score rose from 34.0% to 65.7% on SpaceCast-Bench; its average across six other spatial benchmarks rose from 33.5% to 38.1%, with results varying by benchmark. This shows gains on the tested data, not reliable performance in physical environments.
SpaceCast-Bench is built from indoor scans and programmatic transformations. The object movements and layout rotations do not simulate physical dynamics such as friction or collision, and no robot trials are reported. The authors have released the code and dataset.