EP 06 · How AI Actually Works
How AI Generates Video: Inside Sora and Wan2.1 on a Real GPU
Wan2.1 on a real GPU: 3D causal VAEs, spacetime tubes, 3D RoPE, simultaneous denoising and why a ball vanishes behind a pillar.
TL;DR
Video models such as Wan2.1 denoise a whole clip at once as one block of space and time, not frame by frame. A 3D causal VAE and spacetime tube tokens shrink the attention problem 975,015×, 3D RoPE gives every token a time and screen position, and cross-time attention keeps motion smooth. With no depth axis, objects can still vanish or stall behind obstacles.
- 10.12 → 27.74 dBframe consistency
- 32,760tokens per clip
- 975,015×attention cut
- 90.37%cross-time attention
- 71.4%occlusion wipeout
Summary
Part 6 looks inside video generation using Alibaba's open Wan2.1-T2V-1.3B on an RTX PRO 6000 Blackwell GPU, and uses it to explain the design shared by Sora-style video diffusion transformers.
Generating a video as separate images fails: independent frames had a frame-to-frame PSNR of 10.12 dB, while 17 frames generated together reached 27.74 dB. The model has to treat the clip as one block of space and time, and that block is enormous. Five seconds at 832x480 and 16 fps is 81 frames and 32,348,160 pixels, so full attention over raw pixels would mean 1.046 quadrillion pairs.
Two compression steps make it tractable. A 3D causal VAE shrinks space 8× per side and time 4× while keeping Frame 0 on its own, which is why clip lengths follow 4k + 1. Grouping latents into spacetime tube tokens leaves 32,760 tokens for an 81-frame clip, a 975,015× reduction in attention cost. 3D RoPE then gives each token a time, height and width address.
Unlike an LLM, the model does not work frame by frame: all frames are denoised together over 30 rectified-flow steps, and by Step 15 about 73.9% of the motion path is already set. One probed head on a moving wheel token sent 90.37% of its attention to other time slices. Guidance changes motion speed as well as colour, and pinning Frame 0 turns text-to-video into image-to-video. The weak spot is depth: with no Z axis, a pillar covering a tennis ball overwrote its features, dropping similarity from 0.9760 to 0.2792, so the ball stalled in front.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
Why can't you make an AI video by generating images frame by frame?
Each frame would start from its own random noise with no memory of the last, so the car, the cliffs and the camera angle change every frame. In the article's test, independent frames from Wan2.1 had a frame-to-frame PSNR of 10.12 dB and an average pixel jump of 57.15 levels. Generating 17 frames jointly in a 3D diffusion transformer raised PSNR to 27.74 dB and cut the jump to 4.11.
How does a video model fit a whole clip in GPU memory?
By compressing twice. A 5-second, 81-frame clip at 832x480 has 32,348,160 pixels, and raw attention would need about 2,093 TB for one head. Wan2.1's 3D causal VAE first shrinks height and width 8× and time 4×, turning 97.04M floats into 2.10M. Grouping latents into spacetime tubes then leaves 32,760 tokens and about 2.15 GB of attention scores. On the test GPU a 17-frame clip took 11.67 seconds.
What is 3D RoPE in video diffusion transformers?
It is how each video token knows where and when it is. Wan2.1 splits the rotary position frequencies of each 128-dimensional attention head into 22 pairs for time, 21 for height and 21 for width (axes_dim = [44, 42, 42]). Because rotations encode differences, two token pairs with the same offset in time, row and column get the same relationship wherever they sit in the clip.
How do AI video models keep motion consistent across frames?
Through attention that reaches across time. Probing Layer 21, Head 7 of Wan2.1, a token on a moving car wheel kept just 9.63% of its attention within its own time slice, sending 90.37% to past and future slices, 40.44% of it in a single slice. Because every slice is denoised together over 30 steps, motion is decided early: 31.2% of the trajectory was already fixed by Step 6.
Why does AI video still break physics?
The model only knows screen position and time, [t', h, w], with no depth axis. When a stone pillar covers a moving object, the pillar's texture replaces the object's features at those positions, so nothing records that the object is still behind it. In the article's test, feature similarity for a yellow tennis ball fell from 0.9760 to 0.2792, a 71.4% wipeout, and the ball stalled and distorted in front of the pillar.
What does guidance scale do in AI video generation?
In video, classifier-free guidance affects motion as well as colour. Sweeping the scale on Wan2.1, moving from w = 1.0 to w = 3.0–5.0 made the car move 2.46× faster in optical-flow terms, from 0.811 px to 1.999 px. Pushing to w = 9.0 overshot: clipped, saturated pixels rose to 6.83%, 34× more than at w = 3.0, and the video picked up neon burn-in artifacts.