Inference Engineering
Serving Economics
The only formula that matters
Serving economics reduces to one quotient:
cost per token = GPU-hour price ÷ tokens served per hour
An H100 rents for roughly $2–4/hour. If a node serves 2M tokens/hour, your hardware cost is ~$1–2 per million tokens — before power, networking, and the engineer keeping it alive. Every optimization in this module — batching, quantization, speculative decoding — is a lever on the denominator.
Now invert it: if demand is lumpy and your GPUs sit 60% idle, effective cost nearly triples. Utilization is margin. This is why providers push batching, subscriptions smooth demand, and 'serverless GPU' products exist at all.
insight
The open-source-model pricing wars were mostly *serving* wars. When Llama weights became free, the remaining cost was HBM and GPU-hours — and every serving breakthrough has crashed API prices since.