A Princeton-led research team has trained VideoGen-Agent to route difficult video prompts through retrieval, simulation, generation and verification tools. On its 600-prompt benchmark, the full agent scored 75.6 out of 100 with its original toolset, compared with 56.5 for the same text-to-video generator used alone. These are average evaluation scores from an AI judge, not percentages of fully successful videos.
A single video generator can make a polished clip while missing the instruction that matters. It may invent the steps of an unfamiliar procedure, lose a named subject's identity, violate collision physics or scramble the order of a multi-stage event. VideoGen-Agent treats those failures as workflow problems: one policy decides which supporting tool to call, reads the result and chooses what to do next.
One policy, six workflows
Its six categories use different workflows. For example, it searches text for procedural knowledge and images for identity references. It sends simulated motion to a motion-conditioned generator for physics prompts. For crowded scenes, it checks generated clips with object detection and depth estimation before deciding whether to regenerate. For multi-shot prompts, it generates separate phases and passes the last frame of one segment into the next.
The policy is Qwen3-VL-8B-Instruct, a vision-language model that can read text and images. The researchers first built 24,000 tool-use prompts across the six categories. Teacher models produced example tool-use sequences for 16,000 prompts used in supervised fine-tuning; the remaining 8,000 prompts supported reinforcement learning, where the policy received rewards for valid tool calls, appropriate use of tool outputs and the final video's quality.
This design separates the agent from the generators it controls. The training toolset used Seedance 1.0 for text- and image-conditioned generation, smaller Wan models for reference and motion conditioning, plus search and verification services. A second evaluation toolset replaced the generation backends while leaving the trained policy unchanged.
Learning mattered more than tool access
VABench contains 100 held-out prompts for each category. Gemini 3.1 Pro scores the videos against category-specific criteria, such as procedural correctness, identity preservation or event order. The overall score averages the six category scores equally.
With the first toolset fixed, simply rewriting prompts moved the overall score from 56.5 to 57.3. Giving the untrained policy tool access reached 59.5. Supervised trajectory training raised the score to 69.2, and reinforcement learning added another 6.4 points to reach 75.6. Those controlled comparisons support the paper's main claim that access to tools alone was insufficient; the policy had to learn when and how to use them.
Replacing the generation backends without retraining the policy lifted the score again, from 75.6 to 86.1. The largest gain over the strongest standalone baseline came on ordered multi-shot prompts, where the agent generated phases separately and joined them in sequence. The result also reveals a dependency: a better policy cannot remove the limits of its generators, and a large part of the best configuration's score came from upgraded video models.
In a separate preference test, four raters compared 100 randomly sampled pairs from the upgraded agent and Seedance 2.0. Across the recorded judgments, the agent output was preferred 84.3% of the time. The paper reports the widest preference margins on procedural and multi-shot prompts, the categories where retrieval and sequential generation supplied information a one-pass model lacked.
The same judge scored training and test videos
Gemini 3.1 Pro scored all 600 benchmark outputs using the same category-specific prompts and weights used to reward the agent during training. The authors separately tested judge agreement with human preferences, but the main leaderboard is still produced by a model judge created within the same study. VABench is also a new benchmark from the VideoGen-Agent team, not an independent evaluation.
The system remains limited by generator quality, tool latency and the coverage of its verification signals. Training and testing span six defined workflows, so performance on longer editing jobs or unfamiliar tool combinations is unresolved. The work shows that a learned router can make specialized video pipelines more effective and can accept upgraded backends; it does not establish a general video-production agent that can verify every visual error.
Sources
- Li et al., “VideoGen-Agent: Reinforcing Video Generation Agents,” arXiv:2609.24997 (submitted September 21, 2026)
- Full paper, including workflow design, VABench results, ablations and limitations
- Lead image: generic Blender video-editing workspace by Ogat, not the research system, reused under the free-software license listed on Wikimedia Commons.