A new study suggests image generators can feed details from a partly formed picture back into their internal text representation, sometimes reinforcing objects or attributes that are already wrong. In an arXiv preprint posted Oct. 9, researchers at Stanford, Tsinghua and New York University report that adding their Semantic Feedback Guidance (SFG) to classifier-free guidance (CFG), a standard prompt-guidance method, raised FLUX's GenEval prompt-alignment score from 0.6661 to 0.7479.
The paper studies multimodal diffusion transformers, image generators that let text and image representations exchange information. Within each denoising pass, attention layers can write details from the partly formed image into internal text tokens; later layers can read those tokens while predicting the next image update. The authors illustrate the risk with a prompt for an apple and an orange: if an early image contains only the apple, feedback from that image can preserve the omission instead of helping the model add the orange.
SFG creates a temporary second prediction in which image-to-text feedback is stronger and text-to-image conditioning is weaker, then steers generation away from that prediction. The change is applied during inference; the model weights are not retrained. SFG uses the extra branch only during the first 20% of sampling steps. The authors estimate about 1.2 times the base model evaluations without CFG, or 2.2 times with it. Its in-place variant, iSFG, changes the ordinary conditional pass directly and adds no model evaluations.
On Stable Diffusion 3.5 Medium, CFG+SFG raised GenEval from 0.6890 to 0.7110. The lower-cost CFG+iSFG variant raised GenEval from 0.6890 to 0.7047 and HPSv2.1 from 31.65 to 32.98, compared with CFG alone. GenEval averages six tests of prompt-image alignment, including whether a generated scene contains the requested objects in the requested arrangement. On HunyuanVideo, SFG also raised the reported VBench total from 83.52 to 84.25 for text-to-video and from 89.43 to 90.04 for image-to-video, compared with CFG.
The results are from the authors' benchmark runs, not an independent replication. They use automated alignment and learned preference scores, with no separate human rating study; the paper remains an arXiv v1 preprint. Whether the gains carry over to everyday prompts remains open.
Sources
- Original paper: “Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback”, arXiv v1, posted Oct. 9, 2026; see Sections 2–3 and Tables 2–5.
- Authors' inference-code release for SFG.
- Illustrative image, not from the study: Benlisquare, “Algorithmically-generated AI-generated artwork of a futuristic city,” generated with Stable Diffusion v1.4; used unmodified under CC BY-SA 4.0 via Wikimedia Commons.