CLIP: Contrastive Pretraining

CLIP's mechanism is almost aggressively simple. Encode N images and N captions in the same batch; compute the N×N cosine similarity matrix; train so the N true pairs on the diagonal score high and the N²−N impostors score low. That's InfoNCE — every caption is a label, every other item in the batch is a negative.

This is why batch size was a headline result, not an implementation detail: 32,768 items per batch means every positive is contrasted against 32,767 negatives each step. Bigger batches = harder discrimination = sharper space. Contrastive learning converts compute directly into supervision density.