Alaya Lab put up the PROWBench project page on Oct. 2 for a benchmark that checks whether AI-generated video shows a scene’s specified actions, not just whether its images look convincing. The accompanying paper was posted to arXiv on Oct. 1.
The benchmark starts with a game-like world whose rules and events are recorded by a program. That record tracks which people and objects are present, how they move and when an interaction should happen, including when an entity is outside the camera’s view. Researchers can then compare a generated clip with the world the program actually ran.
PROWBench contains 170 constructed episodes and 600 proxy videos. Each proxy turns the same recorded scene into a different visual guide, from a coarse 3D scene to colored boxes around each entity. The authors tested 11 video models, using five-second clips for the main comparison and smaller tests for longer videos and multiple camera views.
Two new measures ask whether the generated video follows the scene’s event log. Logic-Render Alignment checks if each prescribed action appears in its assigned time window. Interaction Success Rate checks whether the action and its expected end state are visible. Both use a vision-language model as a judge. In one example, a standard image-text similarity score ranks the weaker clip higher, even though it misses the programmed interaction.
In the verified short-clip track, each test began with a first image checked against the scene’s recorded people, positions, facing directions and layout. Across 40 five-second scene-camera pairings, MiniMax-H3 scored 0.675 on Interaction Success and 0.738 on Logic-Render Alignment, both on a zero-to-one scale. The authors also report that it followed camera movement about as well as several world models given an explicit camera path.
The longer test exposed a different problem. Across ten 30-second recordings, a score comparing the box around each person or object in the generated video with its recorded box ranged from 0.175 to 0.280; one means the boxes overlap exactly, zero means they do not overlap. For entities that left the frame and returned, this score was only 0.053–0.131. The benchmark therefore found that a convincing start did not guarantee consistent scene details over time.
The results are early evidence from a constructed benchmark, not a real-world physics test. The longer-video set contains only ten recordings, and the vision-language judge has not been calibrated against human annotations. The project repository lists the benchmark data and evaluation code as upcoming, so outside teams cannot yet rerun the tests from those materials.
Sources
- PROWBench paper (arXiv, Oct. 1, 2026)
- Authors’ project page
- Authors’ code repository and release roadmap
- Lead image: Alaya Lab authors’ overview figure in the paper, reproduced unmodified under CC BY 4.0.