Inside the Transformer
Positional Encoding
8 min+80 XPStep 1/11
Attention has no sense of order
Here's a subtle problem: attention treats the input as a set. Shuffle the tokens and every attention score stays the same — 'Dog bites man' and 'Man bites dog' contain identical tokens.
Nothing in the raw mechanism knows which word came first. Order — the thing that makes language language — has to be injected from outside.
warning
Without position information, 'the cat ate the fish' is indistinguishable from 'the fish ate the cat'. Bag of words in, nonsense out.