Making Models Fit
KV Cache Memory Math
The cache that eats VRAM
During generation, attention would recompute keys and values for every previous token at every step — quadratic waste. The fix: cache each token's K and V vectors once, then read them back. This is the KV cache, and it's what makes autoregressive generation practical.
Its size is pure arithmetic:
bytes/token = 2 (K+V) × layers × d_model × bytes/elem
A 32-layer model with d_model 4096 in fp16: 2 × 32 × 4096 × 2 ≈ 0.5 MB per token. Now multiply: a 128k-token context is ~64 GB — for one request. That rivals the weights themselves.
insight
This is why '1M-token context' products cost more per request: the context isn't just read once at prefill — it must be stored, read at every decode step, and kept resident in the most expensive real estate in computing (HBM).