The Pre-Transformer Era
Why Attention Was Inevitable
8 min+90 XPStep 1/10
Permission to look back
Bahdanau, Cho, and Bengio (2015) made the decoder's wish concrete: keep every encoder hidden state — one per input word — and at each output step, score which input words are relevant right now, then blend those states proportionally.
Translating 'l'accord'? The model stares at 'agreement', not at 'the'. These soft alignments were beautiful — you could watch the model's eyes move — and long-sentence performance stopped collapsing. Attention had arrived, wearing the costume of an RNN add-on.
insight
Those alignment maps were the first time anyone could see inside a translation model — the attention weights doubled as an explanation.