Full fine-tuning updates every parameter: it is expensive and leaves a whole copy of the model per task. Parameter-efficient fine-tuning (PEFT) freezes the model and trains a small add-on. The first were adapters (Houlsby et al., 2019), modules inserted in each block: with 3.6% of BERT's parameters they came within 0.4% of the performance of full fine-tuning (Alammar & Grootendorst, 2024, p. 359).
LoRA (Hu et al., 2022) adds no layers: it learns the correction to each weight matrix W₀, constrained to be the product of two thin matrices, ΔW = BA, with B of size d × r, A of size r × k, and a rank r much smaller than d. The layer computes h = W₀x + BAx. Since B starts at zero, training begins exactly from the original model. In a 12,288 × 12,288 matrix of GPT-3 (≈ 151 million weights), r = 8 leaves about 197,000 (Alammar & Grootendorst, 2024, pp. 362–364). Once training ends, BA is added to W₀, so inference costs nothing extra.
The result is one base model and many swappable modules, one per task (2024, pp. 360–361). QLoRA adds 4-bit quantization of the frozen model (2024, pp. 364–367); this way, Dettmers et al. (2023) fine-tuned a 65-billion-parameter model on a single 48 GB GPU.
Why it works. The hypothesis is that adaptation has a low intrinsic dimension: the needed changes live in a small subspace (Alammar & Grootendorst, 2024, p. 364). In GPT-3, Hu and colleagues found tasks for which r = 1 sufficed, and that ΔW amplifies directions already present in W but not emphasized there. If tuning is mostly amplifying what pretraining left behind, does the model learn anything new, or choose among what it already knew? The authors warn that a low rank may not suffice if the task is in a language other than that of pretraining.