Multimodal Models

A vision-language model like GPT-4V or Gemini is architecturally simple: run a CLIP-style vision encoder over the image, convert its output into a sequence of visual tokens, and interleave those tokens with the text tokens in the LLM's context.

From the language model's perspective, the image is just more tokens to reason over. Attention does the rest — the words 'what color is the car?' attend to the visual tokens that came from the car. That single trick buys chart reading, meme explanation, photo Q&A, and whiteboard transcription.