A video model can make a convincing scene while getting the physics wrong. A new 40-task benchmark found that the top image-to-video system—a model that animates a still frame—averaged 57.76 out of 100 on tests of physical consistency.
The researchers gave every system the same starting image and prompt for each task, then generated four clips per model-task pair, for 1,280 videos in total. The tests covered falling objects, pendulums, collisions, light, liquids, melting, magnetism and surface tension.
Rather than compare each clip with a reference video, the benchmark measures physical relationships in the generated scene. For example, it checks whether a pendulum’s period changes with its length as expected. A video-language model first screens whether the objects and events remain visible and trackable; software then extracts positions, angles, timings or liquid levels for the task-specific physics score.
Seedance 2.5 had the highest average score, but results varied sharply by task. All eight systems scored between 93 and 99 on communicating vessels, where connected liquid levels should settle evenly. On the harder ice-with-stone melting test, all eight systems scored zero for physical correctness; any points in the task’s composite score came solely from a separate visibility-and-continuity screen.
The researchers also compared the metric with human ratings of 320 videos from 10 annotators. On 917 assessable pairs of videos from the same task, the metric’s rankings agreed with human ratings 51.6% of the time, compared with 43.4% for a direct vision-language-model baseline; human raters agreed with each other 56.6% of the time. The automated score beat that baseline but remained below human-to-human agreement.
The result is a benchmark for 40 controlled scenes, not proof that video models can predict what will happen in a real environment. The authors say the evaluation remains limited by whether relevant objects stay visible and measurements can be extracted reliably; the study does not test whether these scores predict robot-planning performance.
Sources
- Mingju Gao et al., “World Models’ Last Exam in Physics,” arXiv:2610.08791v1 (submitted October 6, 2026; CC BY 4.0). Lead image is the paper’s teaser figure, used unmodified.
- Full paper (HTML)