TopolearnLoRA cuts trainable parameters by 100x or more, making LLM fine-tuning feasible on a single GPU.
Full fine-tuning updates every weight and needs memory for gradients and optimizer state for all of them. LoRA (Low-Rank Adaptation) freezes the original weights and learns a small update for selected linear layers: W + (alpha / r) * B @ A, where A and B have a small rank r (8 to 64). Only A and B are trained, usually well under 1% of parameters.
With the PEFT library (pip install peft):
config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(model, config)
model.print_trainable_parameters()target_modulesnames the layers to adapt; module names differ per architecture (q_proj/v_projin Llama-style models,query/valuein BERT)."all-linear"targets every linear layer and often works best.- Use a higher learning rate than full fine-tuning (around 1e-4 to 2e-4).
- The wrapped model works with
Trainerunchanged.
model.save_pretrained(dir) saves only the adapter (a few MB). Load it later with PeftModel.from_pretrained(base_model, dir), or call merge_and_unload() to fold the adapter into the base weights for deployment without the PEFT dependency. Several adapters can be loaded onto one base model and switched per request.
Example
from peft import LoraConfig, PeftModel, get_peft_model
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M")
config = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()
# trainable params: ... || all params: ... || trainable%: well under 1%
# ... train with Trainer(model=model, ...) as usual, lr around 2e-4 ...
model.save_pretrained("smol-lora") # adapter only: a few MB
base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M")
tuned = PeftModel.from_pretrained(base, "smol-lora")
merged = tuned.merge_and_unload() # plain model, adapter folded in
merged.save_pretrained("smol-merged")Remember
- LoRA freezes base weights and trains small low-rank matrices
target_modulesmust match the architecture's layer names (or useall-linear)- Adapters save in MBs;
merge_and_unloadfolds them in for deployment
Watch out for
- Copying target_modules=['q_proj', 'v_proj'] to a BERT model, where those layer names do not exist, which raises an error.