AI assistants that remember past chats often compress them into notes before they know what a user will ask next. Those notes are quick to search, but can leave out an image or detail that becomes important later. In an Oct. 5 preprint, researchers from Nanyang Technological University, Tsinghua University and the University of Illinois Urbana-Champaign describe MemPilot, a system that decides whether to use compact memory or revisit selected parts of the original multimodal history.
MemPilot keeps both versions of the past: a compact memory bank for quick retrieval and the original conversation and image history. For each new question, a small language model chooses between searching the compact notes and asking another model to gather and summarize relevant evidence from the original history. It can set how much to retrieve, the instruction for the summarizer, which language or vision-language model to use, whether images are needed and when to stop. The orchestrator itself is text-only; delegated models handle selected images.
The orchestrator, based on Qwen3-4B-Instruct, was trained with reinforcement learning: it learned from scores that balance answer quality, model-use cost and estimated latency. Training used portions of Mem-Gallery, WorldMemArena and H2HMem, with held-out test sets from those benchmarks; MemEye and MemLens were reserved for evaluation on other data. The paper reports that the performance-focused setting received the highest answer-quality score from an AI judge, GPT-4o-mini, on all five benchmarks with both answer models, Qwen3-VL-4B-Instruct and Qwen3.5-9B. Balanced and cost-focused settings use less token-based inference but score lower on judged answer quality. The authors also publish code under the Apache-2.0 license.
These are author-reported benchmark results, not a test of assistants in production. During training, latency is estimated with a proxy rather than measured as end-to-end wall-clock time; the authors leave deployment-aware latency tests for future work. And because the system can send selected conversation history to external models, private details may leave the service holding the memory. The paper identifies privacy-aware routing or local processing as open needs.
Sources
- MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents (arXiv preprint, Oct. 5, 2026)
- MemPilot code repository
- Image credit: Figure 1 from the MemPilot preprint by Haozhen Zhang et al., CC BY 4.0.