Researchers built Ledger, a 3D memory that records where objects appear in first-person video and how their locations change. It let a language model answer spatial questions from text records without reopening the original images or video.
The system targets questions such as where an item was left or what a container held. Ledger builds its memory offline from sampled recordings, keeping each object's 3D locations, movement history and a short description of context, such as the surface it rested on. It records a move only after repeated observations support the change, reducing the chance that localization noise looks like movement.
When a question arrives, Ledger retrieves relevant object records and gives them to a language model as text; the answerer does not see the source frames in this test. On all 2,400 HD-EPIC questions about 3D perception and object motion, GPT-5.4 answered 29.7% correctly with only the question and options, and 44.6% with Ledger's text memory, counting each question equally. GPT-5.4 ran without extended reasoning; the study used recorded video rather than a live assistant watching a user.
Scene changes remained a weakness. In 100 video streams stitched from multiple scenes, performance fell; building a separate memory for each scene partly recovered the loss, the paper reports.
Sources
- Shravan Chaudhari and colleagues, “Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos,” arXiv:2610.10538v1, submitted October 7, 2026.
- Full paper. Methods, HD-EPIC benchmark, result aggregation and multi-scene analysis.
- Lead image: Figure 1 by Chaudhari et al., arXiv:2610.10538v1, CC BY 4.0.