The Allen Institute for AI (Ai2) released Olmo-core 3 on Oct. 1, 2026, an open training system designed for mixture-of-experts (MoE) language models. These models route each text token to a few specialist components rather than activate every parameter; the new software aims to reduce the GPU-to-GPU data movement that can erode those compute savings.
MoE models can grow their total parameter pool while using only a fraction of it for each token. Training still has to store and update the full pool across GPUs, however, and the tokens must reach the experts selected to process them. If moving weights and data takes too long, the communication overhead can eat into the efficiency gained by activating fewer parameters.
Ai2 says its earlier MoE training path used fully sharded data parallelism (FSDP), gathering and resharing model weights for each small training batch. Olmo-core 3 adds a design based on distributed data parallelism (DDP) that keeps experts resident on GPUs and routes the relevant data to them. This targets repeated weight movement; routing still requires communication between parts of the system.
In a preliminary test on eight NVIDIA B300 GPUs, Ai2 reports that a 47-billion-parameter MoE processed 52,000 training tokens per second per GPU, compared with 19,400 on its earlier implementation. The result is about 2.7 times the throughput in Ai2’s comparison of the two training stacks. It does not isolate the effect of the DDP change or measure whether the resulting model is more capable.
A separate benchmark expanded the expert pool from eight to 128 while routing each token to four experts. Active parameters per token stayed near 3.2 billion as total model capacity rose from 4.6 billion to 47 billion; Ai2 reports that throughput fell by less than 5%. That test shows the stack handling a larger pool without a comparable increase in active parameters per token, not a quality gain from the extra capacity.
Ai2 also tested a configuration with 1.2 trillion total parameters across 512 B300 GPUs. The test used random routing to measure system performance, rather than a trained model’s quality. The public Olmo-core repository gives other researchers access to the training code, but these release figures remain Ai2’s own measurements.
Sources
- Ai2: Introducing Olmo-core 3
- Olmo-core source code on GitHub
- Image: ChrisDag, Wikimedia Commons, CC BY 2.0. The archival photograph shows a Tesla GPU cluster, not Ai2’s B300 test hardware. License.
.jpg){kind=link}