Embeddings for Words

A tokenizer turns 'cat' into integer 7314 — an arbitrary label. The naive encoding, one-hot, makes it a 50,000-dimensional vector with a single 1. Every word is exactly as far from every other word: 'cat' is as distant from 'kitten' as from 'carburetor'.

An embedding layer fixes this with beautiful simplicity: a lookup table, trained end-to-end like every other weight. Each word gets a dense vector — say, 300 numbers — and the network learns where to place them.