A text-to-image diffusion model scored higher on three benchmarks when it predicted clean image features directly instead of the usual flow-matching velocity, a new paper reports. The tests used two image encoders tuned to preserve fine visual detail.
Visual encoders turn pixels into feature representations that image generators can process. Representation autoencoders use those features as a numerical working space, but pretrained encoders may discard small objects, textures and other details. Fine-tuning an encoder to reconstruct its input can recover those details, while also changing the geometry of the feature space.
Fine-tuning narrows feature variation, but velocity targets still include extra noise
In the researchers’ analysis of 200,000 ImageNet image patches, 129 principal components explained 90% of the variance in reconstruction-tuned DINOv2-L features. The frozen encoder needed 672 components for the same share. Although both representations have 1,024 dimensions, most of the fine-tuned features are concentrated in a smaller subspace.
That geometry creates a problem for the standard flow-matching target. Flow matching trains a generator to predict a velocity vector along a defined path between random noise and a clean image representation. Because velocity prediction covers the full path from noise to the image representation, its target includes directions perpendicular to the subspace where most feature variation lies. The paper’s analysis suggests that fitting those directions makes training less efficient.
Predicting clean features improved both encoders
The researchers instead trained the model to predict the clean feature representation directly, then expressed the flow-matching loss in terms of that prediction. The change improved results with both reconstruction-tuned encoders they tested: DINOv2-L and MAE-B-RAE.
GenEval and DPG-Bench are benchmarks for text-image alignment. For fine-tuned DINOv2-L, GenEval rose from 30.96 to 39.48 and DPG-Bench from 67.47 to 71.83 when the model switched from velocity prediction to clean-feature prediction. COCO-30k FID fell from 29.97 to 16.80; lower FID indicates generated images are closer to the reference image distribution.
The same pattern appeared with MAE-B-RAE: GenEval increased from 36.17 to 40.89, DPG-Bench from 67.54 to 73.90, and FID fell from 22.20 to 17.24. Both comparisons used a one-billion-parameter diffusion transformer trained for 90,000 steps at 256-pixel resolution on a 34-million-example subset of the public BLIP-3o dataset.
Text alignment and image fidelity still trade off
The study does not show that clean-feature prediction wins every measure. A version that compressed DINOv2-L features from 1,024 dimensions to 32 scored slightly higher on GenEval and DPG-Bench than the full-dimensional clean-feature model. But its FID was 17.64, compared with 16.80 for the full-dimensional version. In this setup, the compressed model aligned a little better with prompts, while the higher-dimensional model scored better on image fidelity.
The comparisons cover two reconstruction-tuned encoders and do not show whether clean-feature prediction helps other encoders or models.
Sources
- Original paper: “On the Diffusibility of High-Dimensional Latents” (arXiv:2609.28473v1).
- Authors’ project page, which identifies the work as accepted to ECCV 2026.
- Lead image: Linus Zoll’s Visualising AI illustration for Google DeepMind, via Unsplash, used under the Unsplash License.