Limits, Failure Modes & the Frontier

The field's central epistemic trap: an explanation can be compelling and wrong. Early feature-visualization work produced gorgeous, intuitive stories about what neurons 'saw' — and later ablations revealed some were confabulation, narratives the visualization method generated rather than discovered.

This generalizes: any interpretability artifact — attention map, probe, SAE feature label, circuit diagram — is a rendering of the model's computation, and renderings optimize for human legibility, which is correlated with truth but not identical to it. The discipline is to treat every interpretable artifact as a hypothesis that survives causal testing, not a photograph.