CLIP: Vision Learns from Language

An ImageNet classifier can only ever say one of 1,000 words — it's a closed vocabulary. Concepts it wasn't trained on simply don't exist to it.

CLIP (OpenAI, 2021) took a different route: 400 million (image, caption) pairs scraped from the web. Two encoders — one for images, one for text — each map their input into a shared embedding space. Training is contrastive: pull matching pairs close together, push mismatched pairs apart, across every pair in the batch. No classifier head at all — just geometry.