Training a pixel decoder—the component that turns a vision encoder’s features into an image—on random combinations of encoder layers improved image generation in an unguided diffusion test. With the generator and its sampled latent inputs unchanged, replacing only the decoder lowered the generation Fréchet Inception Distance (gFID) from 3.01 to 2.21 on ImageNet-256. Lower gFID means the generated images more closely match the benchmark’s image distribution.
Representation autoencoders turn images into features from a pretrained vision encoder, then use a decoder to reconstruct pixels and a diffusion model to generate new feature patterns. Shallow encoder layers tend to preserve fine visual detail; deeper layers can produce features that are easier for the generator to model. A fixed blend of layers can therefore suit reconstruction better than generation, or the reverse.
FuseReg trains the decoder on changing random subsets of encoder layers rather than a single fixed blend. The decoder learns to reconstruct images from different layer combinations. The researchers also apply the same idea to the diffusion model, training it to predict the full-layer blend from those changing subsets. At inference, the system still uses the full-layer mean by default, and the pretrained encoder is unchanged.
The fixed-generator test used an RAEv2 DiT-XL and a frozen DINOv3-L encoder on class-conditional ImageNet-256, with 50,000 generated samples evaluated over 50 Euler steps. Training both stages with FuseReg also lowered unguided FID from 13.96 to 9.93 for DiT-Base. The best settings differed by model size and evaluation measure.
The results cover one image benchmark, one vision encoder and two generator scales; the authors say transfer to other encoders, resolutions and data domains still needs testing. Their experiments also show that guidance can change the measured gains, so the reported FID improvements do not establish the same result for every image-generation setup.
Sources
- FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders, arXiv:2609.31620 (v1 submitted Sept. 25, 2026).
- Authors’ code, experiment details and checkpoints.
- Figure 3 by the authors, from the paper above; CC BY 4.0.