EP 03 · How AI Actually Works
How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower (Every Number Measured)
A passcode hidden in a 32,768-token book, and the attention heads, memory tricks and position clocks that find it.
TL;DR
Qwen3-4B found a passcode hidden in a 32,768-token book 20 out of 20 times, in 2.04 seconds. It works because FlashAttention never stores the 2,304 GB attention table, grouped-query attention shrinks the KV-cache to 144 KB per token, RoPE encodes distance, and a handful of retrieval heads do the reading while 63.8% of heads park on an attention sink.
- 32,768tokens read
- 2.04 sfull prefill
- 14.26 GBpeak VRAM added
- 98.4%Floor 29, Head 11
- 74.4%KV saved by hybrid layers
Summary
Part 3 hides one sentence, “The secret passcode to the Kesharpur Vault is 7492”, in the middle of a 32,768-token document of dense technical chapters and asks Qwen3-4B to find it on a 96 GB RTX PRO 6000 Blackwell GPU. Across 5 needle depths and 4 context lengths it answered correctly 20 out of 20 times, reading the full book in 2.04 seconds.
The article then explains how. Standard attention compares every token with every earlier one, so 32,768 tokens means 618.49 billion pairs across 1,152 heads, and storing those score tables would need 2,304 GB. FlashAttention computes the exact same result in on-chip tiles; the measured peak memory increase was 14.26 GB. Grouped-query attention, with 32 query heads sharing 8 key-value heads, cuts the KV-cache from 576 KB to 144 KB per token, and DeepSeek's MLA goes further. RoPE encodes position as 64 rotating clock hands with rope_theta of 1,000,000; forcing that clock to wrap dropped retrieval from 100.0% to 0.0%.
Inside the model, 735 of 1,152 heads (63.8%) mostly park on the first 4 tokens and the last 64, an attention sink. Removing the sink from a sliding window sent perplexity to 151.7, against 1.39 with full attention. A few retrieval heads do the actual reading: Floor 29, Head 11 put 98.4% of its attention on the hidden sentence, and switching off 16 such heads made the model guess 8135. That division of labour is why Gemma 3 and Qwen3-Next mix local and global layers; converting 27 of 36 layers to sliding windows cut the 32K cache by 74.4%.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
How can an LLM read a whole book at once?
By never building the full attention table. For a 32,768-token book, Qwen3-4B compares 618.49 billion token pairs across its 1,152 heads, and writing out every score table in bf16 would take 2,304 GB. FlashAttention instead streams small tiles of queries, keys and values through fast on-chip SRAM with an online softmax, giving exactly the same output. On the test GPU, peak memory rose by only 14.26 GB.
What is the KV-cache and why is it so large?
During generation, every token already read leaves a key and a value vector on every layer, kept in GPU memory so they are not recomputed. With classic multi-head attention (32 KV heads), Qwen3-4B would need 576 KB per token, or 18.0 GB for a 32K book. Grouped-query attention lets 32 query heads share 8 KV heads, cutting that to 144 KB per token and 4.5 GB for the same book.
What is an attention sink in transformers?
Softmax forces each head's attention to add up to 100%, so a head with nothing useful to read needs a harmless place to put it, and models use the first few tokens. In Qwen3-4B, 735 of 1,152 heads (63.8%) send more than 80% of their focus to the first 4 tokens plus the last 64. A 256-token sliding window without those sink tokens reached a perplexity of 151.7; adding the 4 sinks back brought it to 18.4.
What are retrieval heads in LLMs?
A small set of attention heads that find and copy facts from long inputs. While answering the vault question, Floor 29, Head 11 of Qwen3-4B put 98.4% of its attention on the hidden sentence, and two other heads were above 96%. Switching off the top 16 retrieval heads, 1.4% of the model, dropped confidence in the first digit from 100% to 25.4%, and the model answered 8135 instead of 7492.
How does RoPE let models handle long context?
Rotary position embeddings split each head's 128 numbers into 64 pairs and rotate each pair by an angle that grows with position, so the attention score depends only on the distance between two tokens. In Qwen3-4B the fastest pair turns 57.30° per token, while the slowest needs 5,063,256 tokens for one full turn, using rope_theta = 1,000,000. With YaRN, Qwen3 stretches its window from 32,768 to 131,072 tokens.