A model can recognize an audio event yet fail to connect it to the scene shown at that moment. A new benchmark tests that link directly and tracks whether people or objects keep the same identity as a video cuts between shots.
OmniCapBench, introduced in an October 8 arXiv preprint by researchers from Tencent’s Hunyuan team and collaborators at Nanyang Technological University and Northumbria University, evaluates nine multimodal models on 786 videos. The clips total 12.8 hours, span 20 broad categories and are all shorter than five minutes.
Instead of asking a model to write one free-form caption, the benchmark requires a structured record of entities, video shots and audio events. Each person or object keeps an identity reference; each sound or line of dialogue gets a time range; and shots connect those records to what is visible at that point. Rules first check whether IDs, timestamps and links make sense. A language-model judge then compares meaning only between units that have already been aligned.
That structure lets the authors separate recognizing a sound from grounding it in the image. On the paper’s reported percentage scale, higher scores indicate better performance on each measure. Gemini 3.1 Pro scored 63.32% on Event F1, which measures whether the model detects audio events, and 51.46% on Event-Shot Association F1, which measures whether it links those events to supporting shots. These are task-specific benchmark results, not general video-understanding accuracy. The model scored 34.43% on Cross-Shot Coreference Consistency, which tests whether an entity keeps one identity across cuts.
A single whole-caption score can overlook a missing link. In a separate audit, identical outputs received holistic scores ranging from 71.24 to 93.92 depending on which language model judged them; OmniCapBench scores for the same outputs ranged from 56.82 to 58.24, a 1.42-point spread.
The results are an author-reported benchmark study, not evidence that a deployed product will fail in the same way. Every clip is under five minutes, and the reference annotations do not include every true detail in each video. The authors also say their annotations began with model-generated drafts, leaving some risk that models from similar families may be favored despite human and programmatic checks.
Image: Figure 1 from the OmniCapBench paper, comparing evaluation approaches. Credit: the paper authors, CC BY 4.0.
Sources