Logit Lens & the Residual Stream

Between the embedding and the unembedding, every layer reads from and writes to a shared channel: the residual stream. Attention heads move information between positions by writing into it; MLPs transform information by writing per-position. Each layer's output is added — nothing is overwritten, only accumulated.

This reframes the architecture: the transformer isn't a pipeline of transformations but a shared workspace where components read a running draft and append their contributions. The stream is the model's working memory, and its contents at layer k are interpretable in exactly one way — as 'the draft so far.'