Fusion Strategies: Early, Late & Cross-Attention

Every vision-language architecture answers one question: where do the streams merge?

• Early fusion — concatenate raw or lightly-processed inputs from all modalities and run one big model over everything. Deepest possible interaction; you pay full joint-attention cost and can't reuse unimodal encoders.
• Late fusion — separate towers per modality; merge only at the end (score averaging, dot product). CLIP-style. Cheap, indexable, deployable per-modality — but the interaction is a single inner product. Nothing nuanced survives.
• Cross-attention — the middle path: separate encoders, then one modality's queries attend the other's keys/values at chosen layers. Interaction happens mid-depth where representations are already semantic.