Skip to content
Generative Models

The Architectures

Autoregressive Models

9 min+90 XPStep 1/10

Generate the next piece, repeat

The third family makes a different bet: factor the distribution into a chain. Don't model 'a whole image' — model p(next piece | all previous pieces). Generate piece 1, feed it back, generate piece 2, feed both back, and so on.

For images the pieces are pixels (PixelRNN, ImageGPT). For audio, samples (WaveNet — the voice of the early Google Assistant). For text, tokens — which is where this family went nuclear.