Features, Not Neurons

The first era of interpretability treated neurons like labeled components: find the neuron that detects curves, the one that detects cats, read the network like a schematic. It failed for a structural reason — polysemanticity. A single neuron routinely responds to utterly unrelated inputs: cat faces and sports cars and certain punctuation. This isn't noise or bad training; it's what the network converges to.

The deep reason: neurons are coordinates, and coordinates are arbitrary. The network isn't obligated to store one concept per neuron any more than a file system is obligated to store one file per disk block. What the optimization actually builds is a set of features — meaningful variables like 'this token is in a quoted string' or 'this entity is a country' — and then encodes them in whatever mixture of neurons is cheapest.