Seq2seq & the Bottleneck

In 2014, Google's Sutskever and colleagues proposed seq2seq: an encoder RNN reads the English sentence and compresses everything into one fixed-size vector — a 'thought vector'. A decoder RNN then generates French purely from that vector.

Context: Google Translate had launched in 2006 on phrase-based statistical translation — giant tables of phrase pairs mined from parallel corpora like UN proceedings, stitching 'le chat' to 'the cat' chunk by chunk. When it switched to neural seq2seq in 2016 (GNMT), output quality visibly jumped overnight. The thought vector had gone to production.