Multimodal AI Systems
One model, every sense
CLIP to LLaVA to any-to-any — how modalities get aligned, fused, tokenized, evaluated, and shipped in production.
Module
Shared Spaces
Why modalities don't meet for free, and the contrastive trick that built the field
Module
Vision-Language Architectures
Early fusion, late fusion, cross-attention — where modalities should meet is the design decision
Module
Beyond Images
Audio, video, and production retrieval — the modalities that don't behave like pictures
Module
Systems & Evaluation
Benchmarks, any-to-any models, and the discipline of knowing when multimodal is the wrong tool
Capstone project
Cross the senses
Use a multimodal model to do something that needs two modalities — caption images, answer questions about a chart, transcribe-and-summarize audio. Test its edge cases.
Deliverable: A demo notebook plus a list of inputs where the model's cross-modal reasoning broke.
Milestones
Finish every lesson to earn the Multimodal AI Systems Certificate.