EP 10 · How AI Actually Works
How Diffusion LLMs Write 64 Words at Once on an A100 GPU
LLaDA-8B against Qwen2.5-7B on an A100: why parallel decoding locks the ending first, mixes up city names in one step, and needs a cache to win.
TL;DR
A diffusion LLM starts from a row of blank mask tokens and fills them in over a few passes instead of writing one token at a time. On an A100, LLaDA-8B fixed structural tokens such as the end marker first. Sampling everything in a single pass gave a 47.45% chance of chimeras like San Angeles, and without a cache, long prompts made it 2.75x slower than Qwen2.5-7B.
- 64tokens filled in parallel
- 47.45%expected one-step San Angeles chimeras
- 0.22%expected chimeras with a second step
- 9.77 vs 26.87tok/s uncached vs AR at 1,024 tokens
- 98.61%prompt states unchanged per step
Video chapters
- 0:00Cold Open: 64 Blank Masks Crystallizing on A100
- 0:43Why 1-Word Decoding Starves GPUs & 15% BERT Fails
- 3:16Inside 16-Step Denoising & dLLM Blueprints
- 6:17The "San Angeles" Crash: Why 1-Step Unmasking Fails
- 8:54Bidirectional Code Infilling & Span-Length Catch
- 10:42The KV Cache Paradox: 2.75x Slower, Then the Fix
- 13:23Complete Diffusion LLM Engineering Blueprint
Summary
Ordinary chat models write left to right, one token per forward pass, so a 7B model on an A100 spends most of its time streaming weights from memory rather than doing math. Diffusion language models take a different route. LLaDA-8B starts with 64 mask tokens after the prompt and, over 16 passes, keeps the predictions it is most confident about and re-masks the rest. This article measures that process on real hardware against Qwen2.5-7B, a matched autoregressive model, and RoBERTa-large as a fixed-ratio baseline.
The order is not random. Position and unmasking step correlate at 0.7593, yet in the very first pass the model locks the word python, the function arguments and the end-of-sequence marker before it writes def. Unmasking in fewer passes breaks things: with one pass, the two slots' own probabilities put the odds at 47.45% that sampling both slots together produces a made-up city like San Angeles or Los Francisco. Adding a second conditioning pass cut that to 0.22%.
Speed is the twist. Because attention looks both ways, changing one token alters every later layer's view of the prompt, so the plain model recomputes everything each pass. At a 1,024-token prompt that took 9.77 tok/s against 26.87 for Qwen. Prompt hidden states stay 98.61% similar between passes, which is why block caching works; an estimate from measured parts puts cached LLaDA near 129.07 tok/s.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
How does a diffusion language model generate text?
It begins with the prompt followed by a fixed number of mask tokens, all blank. Each forward pass predicts a token for every blank slot at once, keeps the most confident few, and masks the others again for the next pass. After enough passes every slot is filled. Because attention is bidirectional, each guess can use words on both sides of it.
Why does a diffusion LLM write San Angeles in one step?
In a single pass each blank slot picks its word without seeing what its neighbour chose. If the first slot leans towards Los and the second towards Francisco, they can combine into a city that does not exist. For LLaDA-8B on our A100, its own probabilities put the chance of this at 47.45% when both slots are sampled at once. With a second pass the second slot sees the first word, and the error fell to 0.22%.
Are diffusion LLMs faster than normal LLMs?
Not automatically. Without a cache, LLaDA-8B must reprocess the whole prompt on every pass, and at a 1,024-token prompt with 64 passes it managed 9.77 tok/s versus 26.87 for Qwen2.5-7B, which is 2.75x slower. Using fewer passes helps: at 16 passes it measured 89.99 tok/s in the step-count test. Real gains depend on caching and confident early unmasking.
Why can't BERT be used to generate text the same way?
BERT-style models are trained with roughly 15% of tokens hidden, so a mostly masked sentence is far outside anything they have seen. At 90% masking, RoBERTa-large reached 11.477 bits of entropy and repeated adjacent tokens 54.73% of the time. LLaDA is trained on every masking ratio from almost none to all, which keeps its predictions calmer at high masking.
How does block KV caching speed up a diffusion LLM?
Between passes, the hidden states of prompt tokens barely move: they averaged 98.61% cosine similarity across all 32 layers. That makes it reasonable to compute the prompt once, keep its keys and values, and run the passes over a small block of 16 tokens. Combined with unmasking every token above 0.90 confidence, generation took about 10 passes in the article's estimate.