KV-Cache & Runtime Memory

Every decode step attends over all prior tokens — rather than recompute their keys and values, the model caches them: the KV cache. Its size is deterministic arithmetic:

bytes/token ≈ 2 × n_layers × d_model × bytes_per_element (× KV heads vs. total heads for GQA)

For a 7B-class fp16 model that's roughly half a megabyte per token — so an 8K context is ~4GB of cache, rivaling the quantized weights. The cache is why 'fits' isn't just about model size: weights + cache + activations + runtime + OS all share the same finite RAM.