A team at Adobe Research and UC Davis has trained OmniSeek, an audio-video AI model, to inspect selected moments in a recording as it answers questions. Instead of relying only on a first pass over the clip, it can request short sound or video segments, add them to its working context and search again when it needs another clue.
The model starts with a coarse view of a video and its synchronized audio. It can then request a specific audio window or a video segment at higher frame rate and resolution, examine the returned clip, and decide whether to inspect another moment. A sound heard at one point can lead it to check a different scene for a matching visual clue.
To train this behavior, the researchers assembled OmniTraj-170K, a set of 169,725 audio-video reasoning examples drawn from 39,797 videos and covering 19 kinds of cross-modal questions. In the final training stage, the researchers gave extra credit for correct answers that became less likely when either the audio or video was hidden. The test is a proxy for whether both streams mattered; it does not guarantee that every answer is truly grounded in both.
On VideoHolmes, a benchmark of 1,837 questions based on 270 suspense short films, OmniSeek scored 74.6%, compared with 59.1% for its Qwen3-Omni-Instruct base model. The 15.5-percentage-point gain followed the full three-stage training process, so the result does not isolate tool use as its only cause. The researchers also tested the system on ten audio-video benchmarks and four general video benchmarks; the latter used standard single-turn answers without tool calls.
Hour-long recordings remain a concern. The paper’s limitations section says the base model has a 32,000-token native context and describes an 88-minute LVOmni example in which context overflow makes the agent stop using tools and invent sensory observations. Its implementation table lists a 65,536 maximum sequence length for later reinforcement-learning stages, but the paper does not explain how that training setting changes the final model’s usable context. The benchmark gains therefore do not establish reliable analysis of arbitrary long recordings.
Sources
- OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning (research paper)
- Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning? (benchmark paper)
- Photo: Film editing workstation, by Marcel Oosterwijk, licensed under CC BY-SA 2.0.
