A proposed GPU memory layout could keep the context of far more paused AI-agent sessions close to the processor while reserving fast memory for sessions generating text. In a trace-driven simulation, adding a flash-memory tier raised resident-session capacity from 165 to about 3,900 per GPU. The design has not been demonstrated on a physical GPU.
Why paused sessions fill GPU memory
An AI assistant keeps a key-value cache, or KV cache, of previous tokens so it does not have to recompute its whole conversation each time it responds. An active session reads that cache at every new token. A session waiting for a tool or a person needs its cache preserved, but rarely reads it until it resumes. Fast high-bandwidth memory (HBM) on a GPU is scarce, so storing every paused session there limits how many conversations a server can retain.
The researchers' design leaves the frequently read caches in HBM and moves older idle caches into a much larger on-package high-bandwidth flash (HBF) tier only when HBM needs space. When a session resumes, its cache is fetched back to HBM before decoding continues. Because old KV blocks do not change, the proposal writes a block to flash at most once on eviction rather than rewriting it each time the session moves between tiers. That matters because flash has limited write endurance and uses more energy per read than HBM.
The simulated gain and its limits
The team replayed software-agent trajectories with Qwen3-Coder-30B-A3B in a simulator that modeled a B200-class GPU with 192 GiB of HBM plus a hypothetical 3 TiB flash tier. At 48 actively decoding sessions, the modeled system spent about 14 milliseconds per generated token and added 0.084 milliseconds to a resumed turn to restore data from flash. The 3,900-session capacity figure counts all retained sessions, including idle ones; it does not mean 3,900 agents were decoding at once.
Those results depend on the paper's assumed flash bandwidth, access latency, energy and endurance, as well as replayed arrival and idle patterns. The study did not measure a built HBM-plus-HBF GPU or a production agent service.
