Skip to content
Multimodal AI Systems

Beyond Images

Video Understanding Is a Systems Problem

10 min+100 XPStep 1/10

Frames lie about time

Here's video understanding's dirty secret: most benchmarks were solvable from a handful of frames. 'Is this a cooking video?' — one frame suffices. The field spent years congratulating itself on static bias: datasets where appearance, not temporal structure, carried the answer.

What's actually hard and rarely measured:

  • State change — 'the glass went from full to empty' requires comparing frames, not reading any one
  • Event ordering — 'what happened right before the alarm' is a temporal join
  • Counting events — 'how many times did she knock' needs duration-aware processing
  • Causality in motion — 'why did the ball roll' requires modeling physics through time

A video model that can't do these isn't watching video — it's reading a contact sheet.

warning

Before trusting a video benchmark, check the single-frame baseline. If a model fed one frame scores within a few points of the video model, the benchmark measured objects, not events.