Stanford researchers have built an AI system that turns intensive-care video and medical records into a searchable timeline of a patient’s hospital stay. It labels actions in one-minute video segments, combines those labels with medication events in hourly reports, and lets a language-model agent retrieve records and relevant clips to answer questions. The team presents the system as a research tool; its paper does not report improved care or patient outcomes.
The Stanford team set out to make changes in patient activity easier to investigate alongside medication records and assessments, which can be spread across long stays and shift changes. TRACER, short for Trajectory Reasoning Across Clinical Event Records, organizes those sources instead of asking a model to absorb days of raw footage at once. Its authors call this setup a “harness”: the records, tools and search process around the language model.
First, a vision-language model examines one-minute video segments and labels 15 clinician-informed actions, such as turning the head, adjusting bedding or interacting with staff. It also writes a short caption. A language model combines each hour’s action records with medication events. When asked a question, an AI agent can search those reports, inspect the electronic health record and retrieve selected video clips as evidence.
On a benchmark of 144 clinician-informed multiple-choice questions about two ICU stays, the TRACER agent using Gemini 3.1 Flash Lite answered 61.0% correctly. With the same model, the strongest baseline setup—a single long prompt containing 475 sampled video frames and the electronic health record—answered 49.3% correctly. The benchmark covered 264 hours of video and records, but only two patients’ stays, so the result measures performance on a narrow research test rather than day-to-day clinical use.
The researchers also ran TRACER’s video-action analysis across 280 patients and about 60,000 camera-hours. It estimated lower frequencies of purposeful activity for the 53 patients with at least one positive delirium assessment than for the 227 without one. That is an association between video-based estimates and an assessment in the records; the study did not test whether TRACER can diagnose delirium or improve treatment.
The authors say the system was not trained on the hospital’s protected records. The records and video were still used during analysis: the paper says they were stored and processed on secure servers with access limited to approved study staff, and cannot be released publicly. The manuscript is a preprint and does not report a prospective test of effects on clinician decisions or patient outcomes.
Sources
Illustrative image: ICU Room, by Navid hameedi, licensed under CC BY-SA 4.0. This photograph is not from the Stanford study.
