Researchers at the University of Rochester and Microsoft have built VideoLoop, a video-question-answering agent that updates its working notes as it searches a long clip. A second AI loop retrieves relevant details from files and rewrites the short memory the video agent carries forward. In a benchmark test, that design improved answers over an agent that only added to its notes, with nearly the same token use.
Long-video agents gather transcripts, frame descriptions and intermediate results as they inspect a clip. When every new detail is appended to one running prompt, useful evidence can be pushed out of view as the context fills. VideoLoop instead saves observations and analysis in a sandboxed file system. After each tool call, a memory orchestrator retrieves material related to the question and revises a bounded working note; the original files remain available for later retrieval.
On VideoMME’s long-video split, which tests questions about extended clips, the authors report that a Gemini 3 Flash agent scored 81.9% with append-only memory and 85.8% with VideoLoop, a 3.9 percentage-point gain. The system used 618.2 thousand tokens per question in that test, about 0.5% more than the append-only agent. The paper also reports gains across four vision-language-model backbones and tests on VideoMMMU and LongVideoBench.
The evidence is still benchmark-based. The paper says the Gemini 3 Flash memory comparison used one run per configuration, so it does not establish how consistently the gain will hold across repeated tests or live video workloads. The authors have released the implementation for others to examine and run.
Sources
- Original research paper: VideoLoop (arXiv)
- Authors’ code repository (Apache-2.0)
- VideoMME benchmark paper
- Cover illustration: Figure 1 by Huang et al., arXiv:2609.38119, licensed under CC BY 4.0.