Quantization

Quantization reduces the numerical precision used to store or compute a neural network's weights and activations. Instead of representing every value as a 32-bit or 16-bit floating-point number, a system may use 8-bit integers, 4-bit integers, or another compact representation. Fewer bits reduce memory traffic and storage, which can make inference cheaper and sometimes faster, especially on hardware with low-precision kernels.

What actually gets quantized

Weight-only quantization changes the stored model parameters but leaves activations in a higher-precision format. This is common for large language models because the weights dominate memory usage. Weight-and-activation quantization also converts intermediate values and can deliver larger speedups, but it is more sensitive to outliers and hardware support. The key-value cache used during autoregressive LLM generation is a separate memory cost; quantizing weights does not automatically quantize that cache.

A rough memory estimate is straightforward. A 7-billion-parameter model stored at 16 bits needs about 14 GB for weights alone. At 4 bits, the raw weight storage is about 3.5 GB, before scale values, metadata, runtime buffers, and the KV cache are added. The real footprint is therefore higher than the simple parameter-count calculation.

Main approaches

Post-training quantization (PTQ) converts a finished model. A calibration set can show the quantizer the typical range of activations so it can choose scales and zero points without retraining the model. PTQ is fast and practical, but aggressive settings can damage a model's accuracy or make errors concentrate on particular layers.

Quantization-aware training (QAT) simulates low-precision arithmetic during training or fine-tuning. The model can adapt its weights to the rounding and clipping errors, so QAT often preserves accuracy better than PTQ when the target precision is especially small. The tradeoff is additional training cost and a deployment pipeline that must match the simulated quantizer.

Quantizers also differ in granularity. Per-tensor quantization uses one scale for a whole tensor; per-channel or group-wise schemes use separate scales for smaller regions and usually preserve accuracy better. Formats such as INT8, GPTQ, AWQ, and GGUF describe different combinations of representation, calibration, grouping, and runtime support rather than a single universal quality level.

Choosing a quantization level

Start with the hardware and the workload. If the model already fits in memory and latency is acceptable, higher precision may be the safer choice. If memory is the constraint, try a moderate reduction first, then measure task-specific quality, latency, throughput, and peak memory. Perplexity alone may miss failures in tool calling, code generation, long-context retrieval, or multilingual output. A smaller model in higher precision can also outperform a heavily quantized larger model once the full system cost is considered.

Quantization is therefore a deployment tradeoff, not a free compression switch: lower precision can make a model usable on more hardware, but the right setting is the one that meets the application's quality and performance requirements.