Skip to content
Audio & Speech AI

Music & Audio Generation

Modern Audio Transformers

9 min+85 XPStep 1/10

The tokenization trick

The fix for the resolution problem came from an unlikely direction: neural audio codecs. Models like EnCodec and SoundStream learn to compress audio into a short stream of discrete tokens — a few hundred per second instead of 24,000 samples — that decode back to near-transparent audio.

Once audio is tokens, everything transfers: a transformer generates token sequences autoregressively, and the codec decodes them to a waveform. AudioLM showed this could continue speech in a speaker's voice; MusicGen wrapped it in a text prompt.

insight

It's the same unlock as language: turn the messy continuous thing into tokens, then next-token-predict. Embeddings and transformers, straight from the LLM track.