Flamingo & Interleaved Context

CLIP-style data is (image, caption) pairs — but real documents are interleaved: text, image, more text, three images, a caption. Flamingo (DeepMind, 2022) trained on exactly that — scraped web pages where images sit inside prose — and the model learned to condition each piece of text on all the preceding visual context, not one image.

That data choice changes the product category: the model isn't an image captioner, it's a language model that can reference arbitrary visual history. 'Like the second image but brighter' means something — there is a second image, in context, to reference.