Induction Heads & In-Context Learning

An induction head completes patterns: given 'The cat sat. The cat __', it predicts 'sat'. Not by grammar — by a concrete mechanism: if the current token X appeared before followed by Y, predict Y.

The implementation needs two layers:

• Layer 1 — previous-token head writes into each token's residual stream: 'the token before me was X.' Now token B's stream carries 'preceded by A.'
• Layer 2 — induction head: the query at the second 'A' asks 'who was preceded by A?' The key at token B answers 'I was.' Attention lands on B, the OV circuit copies B's value forward, and the logits favor B.

It's a pointer algorithm implemented in continuous algebra — and it's learned, not designed.