Shared Spaces
The Modality Gap
Two spaces that don't touch
Train a text encoder and an image encoder separately and each builds a perfectly good internal geometry — 'king' near 'queen', dog photos clustered away from landscapes. But the two spaces are oriented differently: dimension 47 means nothing across the boundary. Cosine similarity between a text vector from model A and an image vector from model B is a random number generator with extra steps.
Alignment is the learned bridge: a training signal that pushes semantically equivalent inputs — this caption, that photo — to nearby points in a shared space. It doesn't emerge from having two good encoders. It has to be trained, on paired data, explicitly.
insight
A 'dog' text embedding and a dog image embedding from independently-trained models live in different universes. Similarity inside a space is structure; similarity across spaces is a project.