TopolearnQuantization lets a 7B-parameter model run in about 5 GB instead of 14 GB of GPU memory.
Weights normally take 2 bytes each in bf16, so a 7B model needs about 14 GB just for weights. Quantization stores weights in fewer bits: 8-bit halves that, 4-bit quarters it, at a modest accuracy cost.
Transformers loads quantized models through a quantization_config:
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True)
model = AutoModelForCausalLM.from_pretrained(repo, quantization_config=bnb, device_map="auto")bitsandbytes quantizes on the fly at load time, needs no calibration, and is the basis of QLoRA. nf4 is a 4-bit type designed for normally distributed weights; the compute dtype is what matmuls run in after dequantization.
Other families:
- Pre-quantized checkpoints (GPTQ, AWQ, compressed-tensors, FP8, MXFP4) load directly from repos that ship them; the config is read from the checkpoint. They are often faster for inference than bitsandbytes.
- torchao offers int8/int4 and float8 quantization integrated with
torch.compile.
Trade-offs: lower bits mean less memory but some quality loss; speed depends on kernels and hardware, and quantized inference is not always faster than bf16. Evaluate on your task. Note the old load_in_8bit=True keyword on from_pretrained is gone; always use a config object.
Example
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
repo = "Qwen/Qwen2.5-7B-Instruct"
bnb4 = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
repo, quantization_config=bnb4, device_map="auto") # needs a CUDA GPU
print(f"{model.get_memory_footprint() / 1e9:.1f} GB") # roughly 5 GB vs ~15 GB in bf16
bnb8 = BitsAndBytesConfig(load_in_8bit=True)
# model8 = AutoModelForCausalLM.from_pretrained(repo, quantization_config=bnb8, device_map="auto")
# pre-quantized checkpoints carry their own config:
# AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct-AWQ", device_map="auto")Remember
- Pass a
quantization_config(e.g.BitsAndBytesConfig) tofrom_pretrained - 4-bit NF4 with bf16 compute is the common QLoRA setup
- Pre-quantized GPTQ/AWQ/FP8 checkpoints load directly and are often faster
Watch out for
- Expecting quantization to always speed up inference; it mainly saves memory and speed depends on kernels.