Researchers at the University of Illinois Urbana-Champaign and Amazon have designed a language model that refines a sequence in parallel while repeatedly checking its token guesses against a shared internal representation. In tests on Sudoku, arithmetic planning and text generation, the method improved on selected diffusion baselines, but it did not beat an autoregressive model on the text-generation comparison.
Diffusion language models start with a noisy sequence and revise many positions over repeated passes, rather than generating one token at a time from left to right. In discrete versions, parallel predictions can miss relationships between tokens chosen in the same pass. Continuous versions keep a shared numeric representation, but may not tie it to a plausible token sequence until the final decoding step.
The researchers' method, called HC-DLM, connects the two processes. At each pass, it reads a provisional token sequence from the shared representation, adds the noise scheduled for the next pass, and uses that version as context when refining the representation. The updated representation then produces fresh token guesses, so a tentative choice can change instead of being locked in early. The team derives a training objective from a lower bound on the likelihood of the token sequence.
On 6-million-parameter models, HC-DLM solved 72.41% of the paper’s harder Sudoku test set exactly, compared with 70.73% for the same-size hybrid baseline CCDD. CCDD did slightly better on the standard Sudoku set, 94.65% to 94.21%. On Countdown arithmetic problems, where a complete valid chain had to reach the target, HC-DLM scored 84.41% with four input numbers and 37.52% with five; CCDD scored 81.18% and 25.35%, respectively. The harder Sudoku set used strategies outside those represented in its training split, and 10% of Countdown target values were held out.
For unconditional text generation on the One Billion Word benchmark, the 118-million-parameter HC-DLM received a generative perplexity of 75.5 when its 1,024 samples were scored by GPT-2 Large. Lower is better. The reported scores were 77.3 for the continuous-diffusion model Plaid and 92.2 for LangFlow, while an autoregressive Transformer scored 66.7. This is a generated-sample fluency measure, not a held-out likelihood test; the paper says it does not report validation perplexity for this flow-based model.
The results test the architecture on moderate-scale tasks, not on a large pretrained language model. The authors list scaling to larger backbones as future work. HC-DLM therefore offers evidence that token feedback can help diffusion models on selected sequence tasks, while its performance at the scale used by deployed language models remains untested.
Sources
- Hui Ren, Zihan Li, Chang Liu, Huidong Liu and Alexander Schwing, “Hierarchical Continuous Diffusion Language Models,” arXiv:2610.02193v1, posted October 1, 2026: original paper (CC BY 4.0).
- Authors’ project page and result tables.
- Image: authors’ conceptual comparison diagram, hosted on the project page: original image, CC BY 4.0.