EP 07 · How AI Actually Works
How AI Actually Learns: Backpropagation, AdamW & Rank-8 LoRA Measured on Qwen3-4B
From one weight to Qwen3-4B on a Blackwell GPU: cross-entropy loss, gradients through 36 layers, AdamW, the full fine-tuning memory bill and Rank-8 LoRA.
TL;DR
Training is one loop repeated: score the next token with cross-entropy, push the error back through every layer, and nudge each weight. On Qwen3-4B, residual connections kept gradients alive through all 36 layers, AdamW beat plain SGD by a wide margin in 25 steps, full fine-tuning would need about 68.87 GB of GPU memory by the standard 16-byte count, and a Rank-8 LoRA adapter learned a new fact in 21.92 seconds.
- 4,022,468,096parameters in Qwen3-4B
- 68.87 GBcalculated memory for full fine-tuning
- 0.0936 vs 3.4469nats after 25 steps, AdamW vs SGD
- 0.41%of the weights trained by Rank-8 LoRA
- 21.92 sfor 40 LoRA steps on the GPU
Video chapters
- 0:00Full Fine-Tuning vs LoRA on a Live GPU (69GB Calculated vs 20s)
- 0:45Part 1: From 1 Weight to a Perceptron, MLP & Weight Matrices
- 3:54Part 2: Logits, Softmax & Cross-Entropy Loss (-ln P)
- 6:16Part 3: Backpropagation, the Chain Rule & Weight Gradients (dL/dW)
- 8:30Part 4: Vanishing Gradients vs. Residual Skip Connections (x + Layer(x))
- 10:40Part 5: Why an 8GB Model Needs 69GB to Train (16-Byte Memory Layout)
- 13:01Part 6: Vanilla SGD vs. AdamW Optimizer (Momentum & Adaptive Variance)
- 15:21Part 7: Forgetting Risk & Rank-8 LoRA Matrices (A & B)
- 18:49Part 8: Live 20s GPU Fine-Tuning, Gradient SVD & In-Place Weight Merging
- 21:34Summary: The Complete Neural Network Learning Pipeline
Summary
This episode starts from a single number. One weight multiplies an input, the error against the target gives a gradient, and one step of gradient descent fixes the output. Add a bias and a non-linear activation and you have a perceptron. Stack those into matrices and you reach Qwen3-4B, whose 4,022,468,096 parameters were counted layer by layer on an RTX PRO 6000 Blackwell. Before training, the model invented an answer about a made-up spacecraft fault code, scoring a perplexity of 79.24 on the true specification.
Learning that fact runs through cross-entropy loss, which punishes confident mistakes far more than mild ones, and backpropagation, which turns the loss into a gradient for every weight by the chain rule. In a controlled 36-layer test, a plain stack lost its gradient almost entirely before reaching the first layer, while the same stack with skip connections kept it strong. On the live model no layer collapsed. In 25 steps on the same adapter, AdamW reached 0.0936 nats while plain SGD stalled at 3.4469.
The catch is memory. With BF16 weights and gradients plus FP32 master weights and two AdamW moments, every parameter costs 16 bytes, so full fine-tuning comes to about 68.87 GB including estimated activations. That figure is calculated, not measured. Rank-8 LoRA freezes the base model, trains 16,515,072 adapter weights in 198.18 MB of state, learned the fact in 21.92 seconds and can be merged back into the base weights.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
How does a neural network learn from its mistakes?
It makes a prediction, measures how wrong it was with a loss, and then asks how each weight contributed to that error. Backpropagation answers this with the chain rule, giving every weight a gradient. An optimizer moves each weight a small step against its gradient, and repeating the loop over many examples slowly lowers the loss.
Why do language models use cross-entropy loss?
Predicting the next token is a choice among 151,936 options, so the model outputs a probability for each one. Cross-entropy takes the negative log of the probability given to the right token. A near-certain correct guess costs almost nothing, while a confident wrong guess costs a lot, which gives the model a strong, well-behaved signal to fix it.
Why are residual connections needed in deep networks?
Without them, the gradient is multiplied by every layer's derivative on the way back, and if each one shrinks it a little, almost nothing reaches the early layers. A skip connection adds the layer's input to its output, so the derivative always contains an identity term. In our 36-layer test that kept the first layer's gradient large instead of vanishing.
Why does fine-tuning a 4B model need so much GPU memory?
Inference only needs the weights, about 8 GB in BF16 for Qwen3-4B. Training also keeps gradients, an FP32 master copy of the weights and two AdamW moment buffers, 16 bytes per parameter in total, plus saved activations. By that standard count, full fine-tuning comes to roughly 68.87 GB, which is a calculation rather than a measured run.
How does LoRA make fine-tuning cheaper?
LoRA leaves the original weights frozen and learns two thin matrices per layer whose product is the update. At rank 8 that is 0.41% of the model's weight count, so the adapter and its optimizer state fit in 198.18 MB. Matrix B starts at zero, so training begins from the exact base model, and the finished adapter can be merged in.