A University of Maryland team reports a way to run a mixture-of-experts language model faster on one GPU by predicting which parts it will need next. In tests with Mixtral-8x7B on a 24GB Nvidia RTX A5000, their system delivered 5.71 times the average output throughput of MoE-Infinity, an existing model-offloading system.
Mixture-of-experts models contain many specialized feed-forward networks, called experts, but route each token through only a few of them. The full set can be too large for a single GPU, so inference software often keeps some weights in CPU memory and transfers them as needed. Waiting for a routing decision before starting each transfer can leave the GPU idle.
The researchers' system, Mira, predicts which experts later layers will need and starts moving them to GPU memory while the current layer is still computing. It keeps frequently used experts in a “hot” cache and puts predicted experts in a separate staging area. Mira stores expert weights in a custom 8-bit format on the CPU; the rest of the model stays in higher precision on the GPU.
On the 24GB test setup, Mira delivered 5.71 times MoE-Infinity's average throughput and 3.86 times HQQ's, another quantized baseline. The researchers tested randomly sampled ShareGPT conversations of different lengths and limited the CPU to eight cores. These figures describe this Mixtral-8x7B setup, not a general speed-up for every model or workload.
The paper also reports similar accuracy to a full-precision baseline on four language tasks, though TriviaQA fell from 0.57 to 0.54. The results are from the authors' September 29 arXiv preprint. At 24GB, the researchers could not run DeepSpeed because it required too much CPU memory, and omitted Fiddler because the test CPU lacked AVX-512.
Sources
- Mira paper and abstract, submitted September 29, 2026.
- Full paper: system design, hardware and benchmark conditions.
- Figure 4 by Sanjali Yadav and Bahar Asgari, under CC BY 4.0; it diagrams Mira's runtime, not benchmark results.