TopolearnQLoRA is the standard recipe for fine-tuning 7B to 70B models on modest hardware.
QLoRA loads the base model in 4-bit NF4 with bitsandbytes and trains LoRA adapters in higher precision on top. The frozen base costs a quarter of its bf16 memory, and gradients flow only into the small adapters, so a 7B model fine-tunes comfortably on a 24 GB GPU.
The recipe:
1. Load the base with a 4-bit BitsAndBytesConfig (nf4, bf16 compute, double quantization). 2. Call prepare_model_for_kbit_training(model) from PEFT. It enables gradient checkpointing, makes inputs require gradients so checkpointing works with a frozen base, and casts layer norms to float32 for stability. 3. Apply LoraConfig with target_modules="all-linear"; adapting every linear layer recovers most of full fine-tuning quality. 4. Train with Trainer using bf16=True, an optimizer such as optim="paged_adamw_8bit" to cut optimizer memory, and a learning rate around 2e-4.
For deployment you cannot directly merge an adapter into 4-bit weights without precision loss. The usual approach is to reload the base model in bf16, load the adapter, merge_and_unload(), and then optionally quantize the merged model again with a method suited to inference.
For supervised instruction tuning, the companion TRL library's SFTTrainer wraps this recipe: it accepts a peft_config, applies chat templates and packs sequences.
Example
import torch
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True)
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-7B", quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model) # grad checkpointing, fp32 norms
model = get_peft_model(model, LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules="all-linear", task_type="CAUSAL_LM"))
args = TrainingArguments(
output_dir="qwen-qlora", per_device_train_batch_size=4,
gradient_accumulation_steps=4, learning_rate=2e-4, num_train_epochs=1,
bf16=True, optim="paged_adamw_8bit", logging_steps=10, report_to="none",
)
# Trainer(model=model, args=args, train_dataset=..., data_collator=...).train()Remember
- QLoRA = frozen 4-bit NF4 base + LoRA adapters trained in bf16
- Call
prepare_model_for_kbit_trainingbefore adding adapters - Merge adapters into a bf16 reload of the base, not into 4-bit weights
Watch out for
- Merging an adapter directly into the 4-bit base and shipping it, which loses precision compared with merging into bf16 weights.