NVIDIA says its Transformer Engine recipe trained Mixtral-8x7B at 9,050 tokens per second per GPU on eight B200 GPUs, compared with 4,096 for a Hugging Face baseline in the company’s benchmark. That is 2.21 times the baseline throughput. The tutorial presents the setup as a route to training biological foundation models, but its measured workload was Mixtral, a language model.
Mixture-of-experts (MoE) models contain several expert networks but send each token through only a small subset. NVIDIA’s recipe replaces a Python loop that runs each expert’s matrix operations separately with GroupedLinear, which batches those operations together and reduces kernel-launch overhead.
The recipe also uses MXFP8, an 8-bit format that applies a scaling value to each block of 32 numbers, and fuses expert-network operations to avoid some conversions and intermediate data writes. These changes target both computation overhead and memory use; the benchmark measures their combined effect.
NVIDIA’s own intermediate run helps show why attribution matters: Transformer Engine in BF16 reached 4,447 tokens per second per GPU, only about 9% above the Hugging Face result. The fastest run also added expert parallelism across eight GPUs and MXFP8. Because the precision and parallel layout changed together, the reported 2.21× gain cannot be assigned to either one alone.
The result is a company benchmark for one Mixtral model on an eight-B200 setup, not a performance result on a biological model. The post reports throughput, not a model-quality comparison, and it does not present an independent reproduction. NVIDIA has published the recipe code for other teams to test.
Sources
- NVIDIA: Efficient MoE Training for Biological Foundation Models
- NVIDIA BioNeMo Mixtral Transformer Engine recipe
- NVIDIA Transformer Engine documentation
- Image: Pokiiri, Nvidia DGX-B200-HGX, CC BY-SA 4.0. The image is an illustrative hardware photo, not a photograph of the reported benchmark run.
