Circuits: Reverse-Engineering Algorithms

A circuit is a subgraph of the network — a subset of heads, MLPs, and their connections — that implements an identifiable algorithm. It's the mechanistic analogue of a subroutine: not a metaphor for computation, but an actual computational structure you can point to, verify, and (in principle) extract.

The Transformer Circuits program decomposed attention heads into two operations:

• QK circuit — where to attend: the query-key interaction that determines routing
• OV circuit — what to write: the output-value transformation that determines what gets moved

This decomposition turns a 'head' from a black box into two analyzable maps — where it reads from, and what it contributes.