At very low bitrates, codecs can lose fine detail. Gen2-VC, described in a Sept. 28 preprint, adds a pretrained video-generation model after an existing codec decodes a clip. The encoded bitstream remains unchanged; instead, the authors test whether a lower-bitrate encode followed by this refinement can match quality scores from VTM-23.0. Their six-dataset comparison reports average gains on perceptual metrics, but the full 1080p pipeline ran at less than one frame per second.
At very low bitrates, a codec has fewer bits available to preserve fine detail, so reconstructed frames can look smooth or blocky. A generative model can restore plausible texture, but it can also change source details. Gen2-VC uses the codec’s ordinary decoded frames as the source anchor for a separate receiver-side refinement step. The existing codec still determines the transmitted bitstream and its reference-frame chain.
The method uses Wan2.1-T2V-1.3B, a pretrained video-generation model, with two small LoRA adapters—trainable add-ons that adjust a frozen model. One adapter refines frames individually. The second processes overlapping groups of frames to use their temporal context. The codec, the model’s video autoencoder and the main generative model all stay frozen. Adapters trained on one learned codec, DCVC-UF, also improved perceptual scores when transferred without retraining to DCVC-RT and three conventional codecs: ECM, VTM and HM.
BD-rate compares the bitrate needed to reach the same quality score across rate-quality curves. Against VTM-23.0, the authors report that Gen2-VC-DCVC-UF matched LPIPS and DISTS scores with average bitrates lower by 86.65% and 94.24%. These metrics compare perceptual similarity, where lower scores are better. At matched PSNR and MS-SSIM scores, which emphasize pixel and structural similarity, the reported bitrate reductions were 12.11% and 28.03%. Results are equally weighted averages across six benchmark sets—UVG, MCL-JCV and HEVC classes B through E—not guaranteed savings on any individual video.
The added computation is substantial. For a 1080p video, the complete pipeline—including the base codec decoder and both refinement stages—ran at 0.342 frames per second on one A100 80GB GPU and 0.557 frames per second on two GPUs. The paper notes that these are throughput measurements rather than per-frame latency, and that its overlapping frame windows add buffering delay. The results therefore show a benchmark improvement, not a real-time streaming system. The authors say they plan to release code and weights after the paper is accepted.
Sources
- Gen2-VC: Unlocking Generative Priors for Video Compression, arXiv:2609.34725 (v1 posted Sept. 28, 2026).
- Wan: Open and Advanced Large-Scale Video Generative Models, the Wan video prior used by the study.
- Microsoft Research’s DCVC repository, for the learned video-codec baseline.
- Illustrative MPEG-1/2 compression-artifact image by Foreade, via Wikimedia Commons, licensed CC BY-SA 4.0. The image is not a Gen2-VC result.
