Quantization

Quantization is a technique that lowers the numerical precision of a model’s weights — for example from 16-bit to 8-bit or 4-bit — to shrink memory use and speed up inference.

Using lower-precision numbers reduces the compute and memory a model needs, lowering inference cost, usually with a small and measurable loss in output quality.

Like distillation, quantization trades some accuracy for cost, so its effect on real tasks should be evaluated rather than presumed negligible.