TopolearnThe same model can be fast or painfully slow depending on a handful of settings.
1. Keep the model on the GPU. Check ollama ps. Anything other than 100% GPU (or full use of Apple unified memory) usually means a big slowdown. Fixes: a smaller model, a lower quantization, or a smaller context.
2. Avoid reloading. Loading a model takes seconds. After a request, Ollama keeps the model loaded for 5 minutes by default. Change this per request with keep_alive ("30m", -1 for "keep forever", 0 for "unload now") or for the server with OLLAMA_KEEP_ALIVE. To preload a model before users arrive, send a request with no prompt, or run ollama run MODEL "".
3. Concurrency. These server settings (environment variables) control many simultaneous requests:
OLLAMA_NUM_PARALLEL(default 1): requests one model handles at the same time. Higher values need more memory, because each parallel slot needs its own context.OLLAMA_MAX_LOADED_MODELS: how many different models may be loaded together (default 3 per GPU, or 3 on CPU).OLLAMA_MAX_QUEUE(default 512): waiting requests before the server answers "busy" (HTTP 503).
4. Memory-saving features. Flash attention, a faster and more memory-efficient way of doing one of the model's core calculations, is used automatically when supported (OLLAMA_FLASH_ATTENTION forces it on or off). The KV cache, the memory that holds the context while generating, can be quantized with OLLAMA_KV_CACHE_TYPE=q8_0, which roughly halves its memory with little quality loss; q4_0 saves more but hurts quality more.
5. Measure. ollama run MODEL --verbose and the API's eval_count and eval_duration fields give tokens per second. Compare settings on the same prompt.
Example
# Load once and keep the model in memory for an hour
curl http://localhost:11434/api/generate -d '{"model": "qwen3:8b", "keep_alive": "1h"}'
# Unload now
curl http://localhost:11434/api/generate -d '{"model": "qwen3:8b", "keep_alive": 0}'
# Server tuning (Linux shell; restart the server afterwards)
OLLAMA_NUM_PARALLEL=4 \
OLLAMA_MAX_LOADED_MODELS=2 \
OLLAMA_KV_CACHE_TYPE=q8_0 \
OLLAMA_KEEP_ALIVE=30m \
ollama serve
# Measure speed
ollama run qwen3:8b --verbose "Explain DNS in two sentences."
# ... eval rate: 61.2 tokens/s
ollama ps # PROCESSOR should say 100% GPURemember
- Check
ollama ps: aim for 100% GPU keep_alive/OLLAMA_KEEP_ALIVEavoid reload delays; empty request preloads- NUM_PARALLEL, MAX_LOADED_MODELS, MAX_QUEUE and KV-cache type trade memory for throughput
Watch out for
- Raising OLLAMA_NUM_PARALLEL on a GPU that is already full, which pushes the model partly onto the CPU and makes every request slower.