An arXiv preprint reports a 2.00× denoising speedup on HunyuanVideo-13B after adding MC-Sparse, a training-free method that skips many attention connections. In a 720p text-to-video test, it was compared with FlashAttention-3 full attention; the timing covers denoising, not the complete generation pipeline.
Video diffusion models refine a clip over many steps. Attention is how one part of a clip draws on information from other parts. Dense attention calculates all these connections; sparse attention uses a selected subset to reduce work. Coarse blocks can keep irrelevant connections and miss useful ones.
MC-Sparse groups similar attention requests into GPU-sized tiles and, at periodic anchor steps, uses full-attention scores to choose individual connections. It records the difference between full and sparse outputs as a correction, then reuses the selections and correction on intervening steps. This avoids repeating the full selection and correction work at every step.
The paper reports a 15% attention-density setting for sparse steps; anchor steps still run full attention to choose connections and calculate the correction. At that setting, the authors measured a 2.00× denoising speedup on HunyuanVideo-13B at 720p and 1.80× on MiniMax-H3-Base at 768p.
On HunyuanVideo, VBench image-quality and background-consistency scores were close to the dense baseline: 68.67 and 96.96, compared with 68.52 and 96.91. The paper’s separate PSNR, SSIM and LPIPS measures compare video output with dense-attention results; these benchmark measures do not report human preference.
The reported speed includes denoising warm-up but excludes condition encoding and VAE decoding. Results are author-reported in an arXiv preprint, and the accompanying repository says its current implementation supports NVIDIA Hopper GPUs only.
Sources
- Original preprint, “MC-Sparse: Deconstructing and Closing the Dense–Sparse Attention Gap in Diffusion Transformers”
- Official project page with method and result comparisons
- Official code repository and hardware support notes
Image credit: Figure 1 from Jiarui Chen et al., “MC-Sparse: Deconstructing and Closing the Dense–Sparse Attention Gap in Diffusion Transformers,” arXiv:2610.06801v1, licensed under CC BY 4.0. The figure is linked to the original paper.