Long conversations, document searches and repeated tool instructions can delay phone-based AI before it starts answering. A new cache design reuses attention data from earlier prompts—even when repeated text appears in a different position—and the authors report a 40% to 60% reduction in time to first token versus no reuse or prefix-only caching.
Before generating words one at a time, a language model processes its prompt in a stage called prefill. It builds a key-value (KV) cache, a set of attention tensors for the prompt, so later generation does not have to recalculate the same context. Reusing that cache across requests can shorten the initial wait when an assistant encounters the same text again.
Most prefix caches help only when repeated text remains at the start of a prompt. But a document assistant may retrieve the same passage in a different order, or a personal agent may reuse tool instructions after a changing chat history. The authors’ design reuses both prefix and non-prefix chunks. To limit answer-quality loss, it selectively recomputes tokens whose cached values differ most from a fresh calculation instead of blindly reusing every matching chunk.
That approach is difficult on a phone because mobile NPUs expect fixed input shapes and precompiled graphs, while useful cache matches vary by request. The system packs the changing work into fixed-size chunks and uses a scheduler to choose among compiled graph calls. Its cache manager places data in NPU memory, phone RAM or flash storage, and overlaps transfers with NPU work to reduce the cost of moving large caches.
On Qualcomm Hexagon NPUs in the Meizu 21, Xiaomi 15 Pro and Honor Magic 8, the team tested four mostly 4-bit models: Qwen3 at 1.7 billion and 4 billion parameters, and Llama 3.2 at 1 billion and 3 billion. Across document question answering, long chat histories and shared agent skills, the system delivered 1.5 to 5.4 times the prefill speed of ExecuTorch without cache reuse and 1.3 to 2.5 times that of prefix caching. These are measurements from the authors’ own implementation, not results for a shipping phone feature.
Reusing non-prefix data can also change how tokens relate to their surrounding context. In two question-answering workloads using Qwen3 on the Xiaomi 15 Pro, the authors report accuracy degradation of 4% or less, but do not say whether that means a relative change or a percentage-point difference. Direct reuse of all matching non-prefix data degraded accuracy by about 30%. That quality result was not measured across every model, device or workload in the study.
The paper compares against the original ExecuTorch engine and an on-device prefix-cache baseline built by the authors. It does not compare against cloud serving systems, which the authors say do not support the tested mobile NPUs. The paper is accepted to EuroSys 2027, scheduled for April. Independent replication across more apps and longer phone sessions will show whether the gains and quality trade-off generalize.
Sources
- Huang et al., “Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs,” arXiv:2609.34727 (submitted Sep. 28, 2026; accepted to EuroSys 2027).
- Lead image: Figure from the same paper, used under its CC BY 4.0 license.