Quantization: The Mechanics

Quantization replaces high-precision values with integers plus a rescaling. For int8: map each float x to round(x/scale) + zero_point, where scale and zero_point are learned during calibration — running representative data through to observe the actual range of values.

Two design axes:

• Granularity — per-tensor (one scale for a whole weight matrix: cheap, crude) vs. per-channel (a scale per row/column: preserves accuracy because channels' ranges differ wildly)
• Symmetry — symmetric quantization (zero_point=0, range ±max) is hardware-friendly; asymmetric fits shifted distributions better

The error introduced is quantization noise — the whole craft is deciding where the network can tolerate it.