Vision in Practice
Video: Vision Plus Time
8 min+85 XPStep 1/10
A stack of frames — with a twist
The naive approach — run an image model on each frame independently — loses the plot. Literally: 'person picks up cup' and 'person puts down cup' contain the same frames in different orders. The meaning lives in the sequence.
Approaches to adding time:
- 3D convolutions — slide the filter over space and time, so it detects patterns like 'edge moving left'
- Per-frame features + aggregation — encode each frame, then pool or recur over the sequence
- Video transformers — patch tokens carry a time coordinate; attention spans space-time directly
analogy
A photo is a word; a video is a sentence. Same letters, but the meaning is carried by the ordering.