Google Research’s September 29 post describes Diffusion Controller, a small add-on network designed to steer image generation toward goals such as closer prompt matching while leaving the base model’s weights unchanged. The researchers evaluated it with Stable Diffusion v1.4.
Text-to-image models build a picture by repeatedly removing noise. Stronger guidance can help an image match a prompt, but can also distort the result. Diffusion Controller adds a trainable side network that adjusts intermediate denoising steps toward a target objective, such as a preference reward, rather than rewriting the base model.
The design is intended for a gray-box setting: the add-on does not need access to the model weights, but it does need the model to expose intermediate denoising outputs. That may make the approach useful when a provider allows access to those outputs while keeping the underlying model closed. It is not a plug-in that can steer any image model without cooperation from its provider.
With HPS-v2, a learned scorer trained to predict preferences from human choices, DiffCon reached win rates of 66.7% under supervised fine-tuning and 68.2% under reward-weighted loss against the pretrained Stable Diffusion baseline. LoRA, an adapter that adjusts some base-model weights, reached 57.7% and 61.1% in the same two setups. The paper reports each method’s best score among several inference-time guidance strengths, rather than at one fixed setting.
The evidence is limited to one base model, Stable Diffusion v1.4, and the gray-box method still requires access to intermediate denoising outputs. The paper does not report tests on closed-source image generators. Personalization and video are future research directions, not demonstrated capabilities.
Sources
- Google Research, “How Diffusion Controller unifies and simplifies AI image generation” (September 29, 2026)
- Tong Yang et al., “Diffusion Controller: Framework, Algorithms and Parameterization,” arXiv:2603.06981 (first posted March 7, 2026)
- Xiaoshi Wu et al., “Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis”
- Image: Benlisquare, Stable Diffusion-generated artwork, CC BY-SA 4.0 via Wikimedia Commons
