Skip to content

LoRA, QLoRA and PEFT

Parameter-efficient fine-tuning trains small adapter matrices instead of all weights, cutting memory and cost dramatically.

Stable: this rarely changes. Reviewed 17 Oct 2026.

Key ideas

LoRA adds low-rank matrices to attention/MLP layers; typical rank 8 to 64.
QLoRA trains LoRA on a 4-bit quantised base model, so 7B to 70B models fit on one or a few GPUs.
Adapters are small files; serve many adapters on one base model (multi-LoRA serving in vLLM).
Tooling: Hugging Face PEFT and TRL, Unsloth, Axolotl; managed options on Bedrock, Vertex and OpenAI.

In code

python

from peft import LoraConfig
cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                 target_modules=['q_proj','k_proj','v_proj','o_proj'],
                 task_type='CAUSAL_LM')

Common mistakes

Overfitting on small datasets; hold out an eval split and watch for regressions on general tasks.

Go deeper