Xiaomi researchers report that a new attention design helped an AI model retrieve information from long conversations and tool outputs more accurately than two related designs. Their calculations also show a smaller cache for retaining earlier tokens. The HySparse2 study, posted September 22, addresses a growing cost of AI agents that repeatedly read large tool responses before producing short replies.
When a model receives another document or tool result, it must process the new text before answering. That input-processing step, called prefill, grows expensive as the history lengthens. Models also keep a key-value cache, a store of earlier token representations used when they generate the next token. HySparse2 changes both where the model spends attention and how its layers share that cache.
Early layers supply data for later caches
In the tested 49-layer model, an initial group of layers processes incoming text and supplies representations from which later layers build their key-value caches. The later layers can then reuse those values instead of processing the full input again during prefill. Within a layer group, sparse attention selects individual older tokens judged useful rather than reserving space for whole blocks; it also always retains a window of recent tokens.
That combination matters for an agent whose relevant evidence may be scattered among many turns. Selecting a block to keep one useful token also keeps its neighbors. Selecting individual tokens spends the same attention budget more precisely, while the shared cache lets the initial layers finish prefill without running the remaining layers on every incoming token.
Higher retrieval scores, with modeled compute savings
The team compared 80-billion-parameter mixture-of-experts models with HySparse2, its HySparse predecessor and a hybrid sliding-window design. They used the same training data and schedule. After additional training on agent tasks, HySparse2 scored 58.45 on the RULER-v2 retrieval test at a 256,000-token context, against 32.61 for HySparse and 35.74 for the sliding-window model. General knowledge and reasoning results were mixed rather than uniformly better.
For a one-million-token context, the researchers calculated a 2.69 GB key-value cache for HySparse2, versus 6.72 GB for HySparse and 12.09 GB for the sliding-window model. HySparse2 also required 2.92 times fewer prefill floating-point operations than HySparse in that calculation. Those one-million-token figures describe modeled storage and arithmetic work, not measured response times; the reported retrieval tests reached 256,000 tokens.
Sources
- Wei and colleagues, “HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing”, arXiv, September 22, 2026.
- Full technical paper, architecture and evaluation, arXiv PDF.
- Illustrative image: server racks at NERSC photographed by Derrick Coetzee, CC0 1.0. The photograph does not show the research team's test system.
