Quantization: fitting giants in RAM

Quantization reduces the numerical precision of model weights (and optionally activations) to decrease memory footprint and increase inference speed. The most common scheme for LLMs is post training quantization (PTQ) to INT8, INT4, or FP8.