Batching: Throughput vs Latency

Remember the decode math: one token requires streaming all the weights from HBM — ~14 GB for a 7B model — to compute a few dozen FLOPs per parameter. At batch size 1, the arithmetic units sit nearly idle; the memory bus does all the work.

Now the key move: batching is free compute. Serve 32 requests together and you read the weights once for all 32 — the same memory fetch, 32× the tokens. The GPU's cost per token falls almost linearly until you finally hit the compute-bound ridge. This is why API pricing exists at all: providers batch strangers' requests to amortize weight reads.