Vision-Language Architectures
LLaVA & the Projector Trick
The cheapest possible bridge
LLaVA's 2023 insight rearranged the field: you don't need to train a vision-language model — you need to connect two that already exist. Take a frozen CLIP vision encoder, take a pretrained LLM, and put a single linear projection between them: image patch features → vectors shaped like the LLM's word embeddings. Feed them as a prefix. Done.
The LLM already knows how to reason over embeddings-in-context; the CLIP encoder already knows how to see. The projector's only job is translation — mapping 'vision features' into 'vectors the LLM treats as words'. One matrix W as the Rosetta stone.
insight
The profound part: it works at all. A linear map preserving enough structure for a 7B model to answer 'what's unusual here' tells you both pretraining regimes learned compatible geometry.