Every parameter of a network is a number stored with some precision. At 32 bits it takes 4 bytes; at 16, two; at 8, one; at 4, half a byte. The arithmetic is simple and decisive: a 7-billion-parameter model weighs about 14 GB at 16 bits and about 3.5 GB at 4 bits, which is the difference between needing a data-center GPU and running it on a personal computer. Quantizing means reducing those bits while trying to keep the important information (Alammar & Grootendorst, 2024, pp. 200–201).
Alammar & Grootendorst (2024) explain it with an analogy: asked the time, we say "14:16," not "14:16 and 12 seconds"; precision is lost, not relevant information (2024, p. 201). As a rule of thumb they advise not going below 4 bits: at 3 or 2 the degradation shows, and a smaller model at higher precision is preferable. Chollet (2021) already described the classic version: converting 32-bit weights to 8-bit integers yields an inference-only model four times smaller and nearly as accurate (2021, sec. 6.3.2).
Large language models brought a complication: from about 6.7 billion parameters onward, feature dimensions with huge outlier values emerge and ruin naive quantization. LLM.int8() isolates them and multiplies them in 16 bits, while more than 99.9% of values are processed in 8 bits, halving inference memory without measurable loss (Dettmers et al., 2022). QLoRA, in turn, quantizes to 4 bits using blocks that respect the distribution of the weights, which makes fine-tuning possible on modest hardware (Alammar & Grootendorst, 2024, pp. 364–367).
Quantization is one of the reasons open weights models such as LLaMA became a household phenomenon, and a nice illustration that a network's knowledge is tolerant of noise: it survives having every weight rounded off.