Precision & Quantization

A parameter is just a number, and a number is just bits:

• fp32 — 4 bytes. Full range, full precision. The old default.
• fp16 — 2 bytes. Half the memory, but a narrow range (max ~65k) that overflows during training.
• bf16 — 2 bytes. Same memory as fp16 but fp32's range — Google Brain's tweak that became the training standard.
• int8 — 1 byte. A lookup of 256 possible values, mapped through a per-tensor or per-channel scale.
• 4-bit — 0.5 bytes. Sixteen values. The frontier.

Do the math once and it sticks: a 70B model is 140 GB in fp16, 70 GB in int8, 35 GB in 4-bit — the difference between a datacenter rack and a laptop.