How Machines Speak
Neural Codecs & Audio Tokens
Compress to understand
Neural codecs — EnCodec, SoundStream, and kin — were built for a boring goal: transmit audio at absurdly low bitrates. Encoder compresses the waveform → residual vector quantization (RVQ) snaps the features to codebook entries → decoder reconstructs the audio. What comes out the middle is the interesting part: audio as a sequence of discrete integers, ~50-100 codes per second.
That representation is the bridge. Integers in a vocabulary are exactly what language models eat — the codec turned 'model audio' into 'predict the next token', and the entire LM machinery became available to speech.
insight
The codec is the unsung enabler of the voice-AI boom. GPT-style modeling on audio was impossible until someone built a tokenizer for sound — and the tokenizer came from a compression lab, not a speech lab.