Continuous diffusion language models return to AI research radar

After a few years of dormancy, research on continuous diffusion language models (CDLMs) has regained traction. That's what Sander Dieleman, a researcher at Google DeepMind (involved in projects like Imagen, Veo, and Nano Banana), documents in a long post published in August 2026. The text is explicitly subjective, a behind-the-scenes account from someone who worked in the field and watched it nearly disappear, but it serves as a map of what's at stake in an alternative to the paradigm that underpins today's LLMs.

What continuous diffusion has to do with language

Modern language models are, in their vast majority, autoregressive: they generate the sequence one token at a time, predicting the next one based on the previous ones. This scheme paired very well with the Transformer architecture, which allows efficient parallel training via teacher forcing, and became the scalable recipe that got us here.

Diffusion starts from a different principle. Instead of building the sequence element by element, the model learns to reverse a corruption process that gradually destroys information, typically by adding Gaussian noise until the signal disappears. The technique has dominated image and video generation. The problem is that Gaussian noise is made for continuous data, and language is categorical: a token is an unordered set, not a continuous physical signal like the light intensity of a pixel.

There are two ways around this. Discrete diffusion (DDLM) adapts the corruption process to the categorical space. Continuous diffusion (CDLM) does the opposite: it represents each token as a continuous embedding vector and applies Gaussian noise there, in vector space, reusing all the machinery already developed for image and video.

The timeline: rise and "extinction"

Dieleman organizes the history into clear phases:

| Period | What happened | |---|---| | 2021 | First discrete diffusions for language: multinomial diffusion, D3PM, SUNDAE | | 2022 | Wave of continuous diffusion: Diffusion-LM, DiffuSeq, SSD-LM, GENIE, and the author's own work (SED and CDCD) | | Late 2023 | "Continuous extinction": nearly all new research shifted to discrete diffusion | | 2024 onward | Gradual revival of the continuous approach |

The appeal of the continuous approach in 2022 was inheriting, for free, the sampling and distillation toolkit from audiovisual diffusion, which was difficult or impossible to apply to discrete methods. But after 2023 the continuous approach vanished from papers, something a 2025 survey shows graphically in the transition from 2023 to 2024.

Dieleman raises hypotheses for the disappearance, always flagging them as speculation. The main one: the "ChatGPT moment" shifted the research focus from theoretical advantages and elegance to raw performance and scale, and the community bet that closing the gap to autoregressive models would be easier with discrete methods, which are conceptually closer to autoregression. A second factor was a hard number: in May 2023, Gulrajani and Hashimoto measured that the continuous model Plaid-1B was 64x less efficient in training than an autoregressive baseline, something hard to take seriously in an era obsessed with compute optimality (Chinchilla).

What makes a CDLM work

The technical part of the post details the ingredients that separate a CDLM that works from one that collapses. Three central decisions:

• Embedding strategy: it can be explicit (one-hot or compact binary patterns like in Analog Bits), pretrained (reusing embeddings from an autoregressive model or a BERT), or learned jointly with the denoiser. Joint learning is the most elegant, but Dieleman warns that naive formulations cause the embeddings to collapse (the denoising error is minimized pathologically when all embeddings become identical) or to grow without control.
• Loss function: MSE when the denoiser predicts directly in embedding space; categorical cross-entropy when it produces probabilities over the vocabulary (which brings the method closer to the autoregressive world, but only works with a vocabulary that isn't too large).
• Noise schedule: for CDLMs this is critical. Since embeddings are high-dimensional vectors representing discrete categories, significant corruption happens within a narrow band of noise levels. Spending model capacity on levels where almost nothing changes (or where everything has already turned to noise) is wasteful.

The author also highlights self-conditioning, a trick that appears in nearly every paper in the field and has a huge impact on performance. And he notes that many of the most ingenious ideas for constraining diffusion to the probability simplex ended up succeeding not in natural language but in biology, where vocabularies are tiny (V=4 for DNA, ~22 for amino acids).

Why this matters to those who build software

The argument underpinning the practical interest is non-sequential generation. An autoregressive model needs N passes for N tokens; diffusion can, in theory, refine the entire sequence in parallel over a few denoising steps, which opens the door to lower latency per response. Dieleman also points to two advantages the continuous approach carries: the ability to represent uncertainty at the individual token level and the rich arsenal of sampling algorithms inherited from image generation.

An editorial caveat is worth noting: none of this is a product available today, and the very history of the 64x gap shows that the promise of efficiency hasn't yet translated into a proven practical advantage at scale. For the Brazilian developer, the immediate value of the text is as a mental map: understanding that token-by-token generation is not the only way to build a language model, and keeping an eye on a front that, if it matures, would change the inference economics of natural-language applications.

What the community said

In the Hacker News thread about the post, one comment summed up the point well:

"'Attention is all you need' should be renamed into 'Attention is sufficient but not necessary'."

>

-- amelius

There was also a counterpoint to Dieleman's historical reading. User janalsncm disagrees that autoregression wasn't yet dominant in 2020/2021: "Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist." For him, what brought chatbots into the mainstream wasn't the architecture but RL (RLHF first, then DPO), and adding diffusion on top of that would be, in his words, "an even bigger pain in the ass".