Researchers at Harvard University and Nvidia have proposed a way to divide long-context artificial-intelligence inference between two types of processor. In an adjusted test using eight Nvidia B200 graphics processors as a hardware proxy, their design produced 31% to 56% more output tokens per joule than the strongest GPU-only configurations. The results appear in a September 11 preprint and have not yet been peer reviewed.
The design addresses a growing mismatch inside large language models. To generate each new token, an attention layer consults information retained from the preceding text, while a feed-forward network transforms the result. Conventional attention may require more work and memory as the context grows. Newer models increasingly combine conventional layers with sparse, linear or sliding-window attention, whose active memory can remain fixed or grow more slowly. Existing split systems still move intermediate results between processors at every layer, so communication can consume much of the expected saving.
The proposed system, called SubQuadratic Disaggregation, or SQD, changes where that transfer occurs. A GPU reads the prompt and keeps the large key-value cache needed by the conventional attention layers. It then sends the current activations to a processor built around fast on-chip static random-access memory (SRAM), where the lighter attention layers and feed-forward networks run together. For sparse attention, the GPU first selects the small set of earlier tokens the model needs; the second processor caches those entries while other layers continue. Data return to the GPU only when the model reaches the next conventional attention layer.
The researchers implemented this sequence in a modified SGLang serving stack on eight B200 GPUs, assigning four to prompt processing and four to token generation. They restricted each generation GPU to 16 of its 148 streaming multiprocessors and adjusted the power calculation to represent a future accelerator without high-bandwidth memory. Tests covered GLM-5.2, Nemotron 3 Ultra and Gemma 4 31B, using chat, retrieval and agent workloads with contexts from 32,000 to 1 million tokens. Performance runs used dummy model weights, while separate traces supplied realistic sparse-attention access patterns.
After the power adjustment, the SQD proxy drew about 4.11 kilowatts, compared with 7.26 kilowatts for the GPU-only system, while retaining 63% to 97% of its token-generation throughput. Tokens per joule improved by 53% for GLM-5.2, 31% for Nemotron 3 Ultra and 56% for Gemma 4 31B. A separate analytical model of Nvidia Rubin GPUs paired with Groq LPX accelerators, checked against the B200 experiment within 15%, projected 1.2 to 1.5 times higher per-user token rates at the same power in latency-sensitive service.
The study does not demonstrate a working Rubin-and-LPX installation or any other physical GPU-plus-SRAM deployment. Its power figures remove components that the proposed accelerator would not contain, several million-token tests exceed the models' stated training lengths, and no production traffic was used. The next evidence will need to run real weights on the intended processors and show that memory transfers, request queues and power measurements preserve the reported advantage under live workloads.
Sources
- Arya Tschand and colleagues, arXiv, September 11, 2026. Paper summary, authorship, submission date and principal results.
- Full paper. System design, execution sequence, hardware proxy, workloads, results and limitations.
- Lead image: Kecko, “CAMPUS - Military Data Center”, used without alteration under the CC BY 2.0 licence. The photograph is illustrative and does not show the experiment.
.jpg){kind=link}