TopolearnFitting the model in fast memory decides whether you get a few words per second or dozens.
A model must be loaded into memory to run. Where it fits decides the speed:
- GPU memory (VRAM): the graphics card's own memory. Fastest.
- Unified memory on Apple Silicon Macs: shared by CPU and GPU, also fast. A 16 GB Mac can use most, but not all, of it for models.
- Normal RAM with the CPU: works, but often 5 to 20 times slower.
If a model doesn't fit in VRAM, Ollama splits it between GPU and CPU. ollama ps shows this, for example 48%/52% CPU/GPU. A split usually means a big slowdown.
Quantization is how big models fit at all. Each parameter is a number. Stored at full precision it takes 2 bytes. Quantization stores it with fewer bits, like saving a photo as a smaller JPEG: a little detail is lost, but the file shrinks a lot.
- 16-bit: original quality, 2 bytes per parameter.
- 8-bit (Q8): about 1 byte, nearly no quality loss.
- 4-bit (Q4, such as
Q4_K_M): about 0.5 to 0.6 bytes, a small loss; the usual default in the Ollama library.
A rough rule: model size ≈ parameters × bytes per parameter, plus room for the context. An 8 billion parameter model at 4-bit is about 5 GB, so it fits a GPU with 8 GB. The same model at 16-bit needs about 16 GB. ollama ls shows real sizes.
Ollama supports NVIDIA GPUs, AMD GPUs (through ROCm), Apple's Metal, and other GPUs through Vulkan. Without a supported GPU it uses the CPU.
Rule of thumb: a smaller model fully on the GPU usually beats a bigger one split onto the CPU.
Example
# Rough memory estimate for model weights (context needs extra memory on top)
def weights_gb(billion_params: float, bits: int) -> float:
return billion_params * bits / 8 # 1e9 params * bits/8 bytes = GB
for name, b in [("4B", 4), ("8B", 8), ("27B", 27)]:
print(name, f"4-bit ~{weights_gb(b, 4):.0f} GB,", f"8-bit ~{weights_gb(b, 8):.0f} GB,",
f"16-bit ~{weights_gb(b, 16):.0f} GB")
# 4B 4-bit ~2 GB, 8-bit ~4 GB, 16-bit ~8 GB
# 8B 4-bit ~4 GB, 8-bit ~8 GB, 16-bit ~16 GB
# 27B 4-bit ~14 GB, 8-bit ~27 GB, 16-bit ~54 GB
# (real 4-bit files are a bit larger, e.g. ~5 GB for 8B, because some layers keep more bits)Remember
- Fastest when the whole model fits in GPU or unified memory
- Quantization stores parameters with fewer bits; 4-bit is the common default
- Size ≈ parameters × bits / 8, plus memory for context
Watch out for
- Ignoring a CPU/GPU split in
ollama psand blaming the model for being slow.