EP 05 · How AI Actually Works
How Vision-Language Models Actually See Images: Inside ViT, CLIP, SigLIP & 3D M-RoPE
Why an “iPod” sticker fools CLIP, and how Qwen2.5-VL turns pixels into tokens a language model can read.
TL;DR
A vision-language model slices an image into patches, encodes them with a Vision Transformer, merges them into a few hundred tokens and feeds those to a language model with 3D position information. That is why Qwen2.5-VL reads the “iPod” sticker as text on an apple, while CLIP, which compresses the image to one vector, calls the apple an iPod at 64.51%.
- 64.51%CLIP says iPod
- 1,600 → 400patches → tokens
- [16, 24, 24]M-RoPE t / h / w
- 15 → 12circles counted
- −41.5%fork hallucination, VCD
Summary
Part 5 asks how vision-language models see, starting from a famous failure. Put a paper label reading “iPod” on a Granny Smith apple and OpenAI's CLIP calls it an iPod at 64.51%; a “Pizza” label wins at 80.07%. Qwen2.5-VL-3B instead says it is an apple with a sticker that reads “iPod”. The article measures why, layer by layer, on a 96 GB Blackwell GPU.
CLIP squeezes the whole image into one vector and multiplies cosine scores by 100.0 before softmax, so a small edge for the printed concept becomes a confident wrong answer. Google's SigLIP, which scores each image and caption with a sigmoid, kept the apple ahead (40.29% against 0.97% for iPod).
A VLM works differently. Qwen2.5-VL cuts a 560 × 560 image into 1,600 patches of 14 × 14 pixels, refines them in a ViT with 4 global-attention layers, then merges each 2×2 group so the language model sees 400 tokens. Keeping the native aspect ratio matters: a wide receipt squashed to 140 × 140 became 25 tokens and the PIN 4829 was misread as 4857, while the native 1120 × 280 version was read exactly. 3D M-RoPE splits position into 16 time, 24 height and 24 width frequency pairs, letting a 1D language model track rows, columns and time.
The article also shows where VLMs still fail: shown 15 overlapping circles, the model counted 12. It ends with Visual Contrastive Decoding, which subtracts the prediction made on a noisy copy of the image; on a plate-and-knife photo it cut the false “fork” probability by 41.5% on a degraded image.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
What is a typographic attack on CLIP?
It is fooling an image model with written words. CLIP learned to read text in photos, so a printed label drags its single image vector toward that concept. Since CLIP multiplies cosine scores by 100.0 before softmax, a tiny edge becomes a confident verdict: an apple with an “iPod” label scored 0.2701 for iPod against 0.2641 for apple, giving iPod 64.51%. Qwen2.5-VL described the same photo as an apple with a sticker.
What is the difference between CLIP and SigLIP?
Both match images to text, but they score pairs differently. CLIP applies a softmax over candidate captions after scaling cosine similarity by 100.0, so the options compete. SigLIP scores each image-text pair independently with a sigmoid, using a learned scale of 117.33 and a bias of -12.93. In the article's test SigLIP resisted the “iPod” label, keeping apple at 40.29% against 0.97%, which the article credits mainly to its learned representations and training recipe.
How many tokens does an image use in a vision-language model?
It depends on resolution. In Qwen2.5-VL, a 560 × 560 image is cut into 14 × 14-pixel patches, giving 1,600 patch vectors of 1,280 numbers each. Before reaching the language model, a 2×2 merger joins neighbouring patches, so the image becomes 400 tokens of 2,048 dimensions. The article's wide 1120 × 280 receipt also produced 400 tokens at native resolution, while a 140 × 140 version produced only 25.
Why do vision-language models miscount objects?
Dense, overlapping objects are a known weak spot. Shown diagrams with 4, 8 and 15 overlapping circles, Qwen2.5-VL-3B answered 4, 6 and 12. A likely contributor is the 2×2 patch merger: each token the language model receives covers a 28 × 28-pixel region, so several ring crossings can be blended into one token before counting starts. The article suggests a detection tool or tiled crops once more than about 5 overlapping items matter.
Why keep the original aspect ratio for OCR with a VLM?
Because squashing an image throws away the detail that text needs. Qwen2.5-VL slices images at their own shape and resolution. A 1120 × 280 shipping manifest read at native size gave 400 visual tokens and the correct vault PIN, 4829. Shrunk to a 140 × 140 square, with 16× fewer pixels, it gave just 25 tokens and the model read 4857 instead. For documents, the article advises setting min_pixels and max_pixels deliberately.
How does visual contrastive decoding reduce VLM hallucinations?
It runs the model twice, once on the real image and once on a heavily noised copy that mostly reflects the language prior, then subtracts the second set of scores. Asked whether a fork was on a table holding only a plate and a knife, the noised image pushed P(“Yes”) to 29.42%. With contrastive decoding that probability fell from 18.24% to 10.67% on a degraded image, and to 0.04% on the clean one. It doubles inference cost.