Speech-to-Speech Models

The classical voice agent is a relay race: ASR → text LLM → TTS. Three models, three latencies, and a brutal information loss at the first baton pass — everything except words dies at the transcript.

Speech-to-speech (S2S) models skip the relay: audio in, reasoning happens over audio representations, audio out. The promise is real:

• Paralinguistics preserved end-to-end — the model hears hesitation and responds with appropriate tone; 'I'm fine' said flatly lands differently than said warmly
• Latency collapses — no intermediate text serialization; responses can begin while the model is still 'listening'
• Non-lexical audio works — laughs, sighs, 'umm', background context: the model can hear and produce what has no transcript