AGENTS 01 · How AI Agents Actually Work
What Happens in 1 LLM API Call? Measured on GPU With Qwen3-8B
Qwen3-8B on an A100, one request at a time: why the model forgets, what a reused prefix saves, why the template matters and why output tokens are slow.
TL;DR
A single request to Qwen3-8B on an A100, taken apart stage by stage. The model kept nothing between calls, so a 10-turn chat cost 5,549 prefill tokens when the history was re-sent and 999 when the shared prefix stayed cached. Without its own chat template it gave 0 of 5 clean answers. At a 1,024-token prompt, output tokens came out 494x slower than input tokens, and a plain request for JSON broke json.loads() in 80 of 80 calls.
- 5,549 vs 999prefill tokens over 10 turns, re-sent vs cached
- 0 of 5clean answers without the chat template
- 494xslower per output token at 1,024 input tokens
- 80 of 80naive JSON replies that json.loads() rejected
- 151,934tokens masked to -inf at step 0
Video chapters
- 0:00Cold Open: Zero Memory & The 494x Speed Gap
- 0:55Stateless Memory & Prefix KV Caching
- 3:08Chat Templates & Control Token ID 151644
- 5:14Prefill vs. Decode on GPU: 7,672 vs 15.5 tok/s
- 7:41Logits, Temperature & Why Naive JSON Crashes 80/80 Times
- 9:51Constrained Decoding: Masking 151,934 Tokens to -inf
- 12:04Production LLM API Blueprint & Bridge to Tool Calling
Summary
Every agent framework is a loop around one stateless request, so this episode measures that request on its own: Qwen3-8B in bfloat16 on one A100, Hugging Face transformers, eager attention, batch size 1. The weights never change during inference, which is why a second, isolated call no longer knew the Postgres port from the first one and guessed the default 5432. Memory is the client re-sending the transcript. Over a 10-turn chat that meant prefilling 5,549 tokens, against 999 when the server kept the key and value tensors of the shared prefix. At Turn 10 the first token arrived in 64.2 ms instead of 139 ms. One catch: Qwen3's template rewrites earlier turns, which breaks the match 4 tokens early.
The template itself matters more than it looks. Formatted with apply_chat_template, five short factual questions got five clean answers that stopped on their own. As a raw string, or wrapped in Llama-2 tags, none of them did: the model kept writing until the 32-token cap. On one prompt, the final position spent 59.1% of its attention on the special-token positions, peaking at 91.4% in layer 30.
Speed is lopsided. Prefill handled 1,024 tokens at 7,672 tokens/s, while decode ran at about 15.5 tokens/s, so a long reply dominates the call. For JSON, a naive prompt wrapped every answer in a Markdown fence, a strict system prompt passed, and a grammar mask that sets illegal tokens to -inf made bad output impossible.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
Does an LLM remember earlier API calls?
No. The weights are read-only while the model runs, so nothing from one request is stored for the next. In our test an isolated follow-up call to Qwen3-8B answered with the generic Postgres port 5432 instead of the 5439 it had just been told. Chat apps feel continuous only because the client sends the whole conversation again with every new message.
What does prefix KV caching save in a multi-turn chat?
When consecutive prompts start with the same tokens, the server can keep their keys and values and only process the new part. Over a 10-turn conversation this cut the prefill work from 5,549 tokens to 999, and the wait for Turn 10's first token fell from 139 ms to 64.2 ms. Keep the system prompt and tool definitions fixed at the top so the shared prefix stays long.
Why do I need the model's own chat template?
The model was trained to see special tokens such as im_start and im_end around each turn. With the native template, five short questions got five clean answers that ended by themselves. A raw string or Llama-2 style tags gave zero clean answers: the model either continued the text like a document or invented new turns until it hit the token limit.
Why is generating output so much slower than reading the prompt?
Prefill pushes all prompt tokens through each weight matrix together, so every weight read is shared by many tokens. Decode makes one token per pass and has to read all 15.3 GiB of weights each time. At a 1,024-token prompt that worked out to 7,672 input tokens/s against 15.5 output tokens/s on our A100, a 494x gap per token.
How do you guarantee valid JSON from an LLM?
Asking nicely is not enough. A naive prompt failed strict parsing in 80 of 80 calls because each reply arrived inside a Markdown code fence. A strict system prompt passed on this small schema, but it only changes probabilities. A grammar mask sets every token that cannot continue valid output to -inf, 151,934 of 151,936 at the first step, so invalid text cannot be sampled.