The Superposition Hypothesis

A network wants to track far more variables than it has neurons: tens of thousands of languages, millions of entities, syntactic states, social cues, physical relations. A 4,096-dimensional residual stream cannot dedicate a dimension to each — so it doesn't.

Superposition stores n features in m < n dimensions by giving each feature a direction and tolerating the fact that these directions can't all be orthogonal. Each feature bleeds a little into every neuron — and into every other feature. The result is a compressed encoding: what information theory permits and gradient descent actually finds.