Quantisation

Storing weights (and sometimes activations and KV cache) in fewer bits to cut memory and speed up inference.

Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.

Key ideas

• FP8 is close to lossless on modern GPUs and widely used in production.
• INT4 methods (AWQ, GPTQ) and GGUF Q4/Q5 make large models fit on small hardware with some quality loss.
• Always re-run your evals after quantising; losses show up first on reasoning and long-context tasks.