Extreme Quantization: GGUF & Friends

Take int8 methods to 4 bits and accuracy collapses — 16 levels can't express weight distributions that matter. Two failure sources dominate: weight outliers (a few important weights dominate the error) and activation outliers (spikes in a few channels force a scale that starves everything else).

The sub-8-bit literature is a series of answers to these outliers:

• LLM.int8() — keep the outlier dimensions in fp16, quantize the bulk to int8; mixed precision targeted exactly where outliers live
• SmoothQuant — migrate the activation outliers into the weights: rescale activations down and weights up by the same factor, moving the quantization difficulty to the side that handles it better