Huawei researchers have simulated a compact computer that stores the weights of trillion-parameter AI models in six high-bandwidth flash packages. The packages supplied 2.30 terabytes per second to the processor, yet all six tested model-and-workload combinations finished within 10% of a reference system with much more exposed memory bandwidth. The September 14 paper is an unreviewed preprint, and the computer has not been built.
The design targets mixture-of-experts models. These models contain many specialist neural networks but select only a few of them for each piece of generated text. The selection reduces computation, while the machine must still store every expert's weights. The two models in the study occupied 865 gigabytes and 1.56 terabytes after compression, far beyond the memory capacity of a typical accelerator.
The proposed computer separates stored model weights from the memory that changes while a request is running. Flash holds all expert weights. When the model's router chooses an expert, only the required pages move to a central processor. Conventional memory holds the key-value cache, which records information from earlier tokens. Separating the two jobs lets designers provide high capacity for weights without giving every memory tier the same costly bandwidth.
The researchers evaluated this arrangement in LLMServingSim 2.0, a simulator for language-model servers. They replayed three multi-step AI-agent workloads on two trillion-parameter models, with eight sessions running at once. Completion time was compared with a reference that exposed 6.14 terabytes per second of host bandwidth and used high-bandwidth memory for the changing runtime state.
Six flash packages connected through 2.30 terabytes per second of total host bandwidth met the paper's target in every test: each workload took no more than 10% longer than the reference. The simulation also found that the changing state needed far less bandwidth than the model weights. That result supports the paper's central design choice of sizing the two memory paths separately.
The reported gains depend on simulated processor speed, flash behavior, cache hits, expert selection and speculative decoding. The paper leaves chip placement, signal integrity and power delivery for later engineering. A physical prototype must run live traffic and report response delay, token speed, energy use, temperature and read errors before the proposed memory balance can be judged as a practical server design.
Related coverage
Sources
- Pengfei Xia and colleagues, “Trillion-Parameter MoE in a Box: Decoupling Memory Provisioning with High-Bandwidth Flash,” arXiv, September 14, 2026. Abstract, authorship, submission date and principal results.
- Full paper. Architecture, model sizes, workload sequence, simulator settings, results and limitations.
- Lead image: NIST Daffodil memory-device test platform, credited to NIST and reused under NIST's public-information guidance. The image is illustrative and does not depict Huawei's proposed appliance.