EP 09 · How AI Actually Works
How Mixture-Of-Experts Runs 35B AI on 3B Of Math
OLMoE's routers traced live on an A100: where MoE weights sit, three frontier blueprints, router collapse, expert blocking and the batch-1 speed catch.
TL;DR
Mixture-of-experts models keep tens of billions of parameters in memory but run only a few billion per token. Measured on OLMoE-1B-7B on an A100, a router sends each token to 8 of 64 experts per layer, early layers lean on one generalist expert, blocking the chosen experts breaks code and calculus answers, and at batch 1 cutting math 5.4x bought only a 1.12x speedup.
- 93.11%weights in expert FFNs
- 8 of 64experts per token
- 1.282B / 6.92Bactive / total params
- 99.83%P(sort) before blocking
- 5.4x → 1.12xless math vs speedup, batch 1
Video chapters
- 0:00Cold Open: 56 of 64 Experts Stay Dark & Live Top-8 Surgical Blocking on A100
- 1:021 · Inside One Layer: Why 93.1% of Weights Live in the Feed-Forward Network
- 3:332 · Three Frontier Blueprints: Qwen3-30B vs. Google Gemma 4 26B vs. Qwen3.5-35B
- 6:203 · Do Experts Actually Specialize? Live A100 Traces at Layer 0, Layer 7 & Layer 15
- 9:124 · Router Collapse (8/16 Dead) & DeepSeek-V3's Aux-Loss-Free Bias Dial (Toy)
- 11:565 · Live Neural Surgery: Blocking Active vs. Unused Experts Across Code, Math & Facts
- 14:396 · The Hardware Catch: Why 5.4x Less Per-Token Math Ran Only 1.12x Faster in HF Eager
- 17:167 · Complete MoE Engineering Blueprint: Why Stored Capacity ≠ Used Compute
Summary
Mixture-of-experts (MoE) models separate what a language model keeps in GPU memory from the arithmetic it does for each token. This article loads Allen AI's OLMoE-1B-7B on an NVIDIA A100, watches its routers live, and compares the design with Qwen3-30B-A3B, Google Gemma 4 26B-A4B and Qwen3.5-35B-A3B, which all land near 3B to 4B active parameters.
Counting OLMoE's 6.92B weights shows why experts replace the feed-forward block rather than attention: the expert layers hold 93.11% of the model, attention 3.88% and the routers just 0.03%. Each token wakes 8 of 64 experts per layer. Gemma 4 adds one always-on shared expert three times wider than a routed one, while Qwen3.5 cuts each layer into 256 micro-experts plus a shared one.
The routing traces do not match the textbook picture. At layer 0 a single expert, Expert 6, takes most Python, calculus, SQL and chat tokens alike. By layer 7 the three technical prompts share most of their top-8 experts (Jaccard 0.60 to 0.78) while the chat sentence shares none. Blocking the chosen experts mid-sentence pushes “sort” from 99.83% to almost zero, yet “Tokyo” stays the top guess.
A toy 16-expert run shows router collapse, with 8 of 16 experts dead by step 150, and how DeepSeek-V3's bias dial reaches lower error than an auxiliary loss. Finally, running all 64 experts took 5.4x more math but was only 1.12x slower at batch 1 in eager Hugging Face code: memory holds everything, compute touches a slice.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
What is a mixture-of-experts (MoE) model?
An MoE model replaces the single feed-forward block in each transformer layer with many smaller expert networks and a tiny router that picks a few of them for every token. In OLMoE-1B-7B each layer has 64 experts and a token uses 8, so only 1.282B of its 6.92B parameters do arithmetic for any given word. The rest wait in GPU memory.
Why does MoE split the feed-forward network instead of attention?
Because that is where the weights are. Measured on OLMoE-1B-7B, the expert feed-forward layers hold 93.11% of all parameters, while the 16 attention layers together hold 3.88% and the routers 0.03%. Splitting the largest block saves the most arithmetic per token, and attention, which moves information between tokens, stays dense and shared.
How do Gemma 4 26B and Qwen3.5-35B differ as MoE models?
Gemma 4 26B-A4B pairs 128 slim routed experts with one always-on shared expert three times wider and runs 3.8B active parameters. Qwen3.5-35B-A3B goes finer, with 256 micro-experts plus a small shared one and 3.0B active, and swaps three of every four attention layers for Gated DeltaNet. Qwen3-30B-A3B uses 128 routed experts and no shared expert.
Do MoE experts specialise by topic?
Not the way the name suggests. In OLMoE on an A100, one expert at layer 0 handled 86.7% to 97.7% of the tokens in Python, calculus, SQL and chat prompts. By layer 7 the three technical prompts shared most of their top-8 experts, while an everyday English sentence shared none. These were four short prompts, so treat it as a small sample.
What is router collapse in MoE training?
Without load balancing, experts that win a few early tokens get more gradient, improve faster and take over. In a toy 16-expert run on the A100, 8 experts had zero load by step 150. A heavy auxiliary loss kept every expert alive but raised task error, while DeepSeek-V3's bias dial, which only nudges expert selection, gave the lowest error of the three runs.
Is a mixture-of-experts model faster than a dense model?
Per token it does far less math, but at batch size 1 the gain can almost vanish. Running all 64 OLMoE experts took 5.4x more arithmetic than the native 8, yet a 30-token pass was only 1.12x slower in eager Hugging Face code. Savings appear with large batches and fused MoE kernels, and every parameter still needs GPU memory.