TopolearnA tool that is great on one laptop needs deliberate planning to serve a team or an application reliably.
Ollama can serve a small team or an internal app well. Doing it reliably takes some planning.
Capacity. Throughput depends on GPU memory and speed. Memory is roughly: model weights + (parallel slots × context length × per-token KV-cache cost). Test with realistic prompts: measure tokens per second and time to the first token while sending several requests at once, then pick OLLAMA_NUM_PARALLEL and the context length that keep everything on the GPU. Requests beyond the parallel limit wait in a queue (OLLAMA_MAX_QUEUE); clients should handle HTTP 503 "busy" replies by retrying later.
Keep models warm. Set OLLAMA_KEEP_ALIVE to keep production models loaded, limit OLLAMA_MAX_LOADED_MODELS, and avoid letting users switch between many models on one GPU; every switch means a slow reload.
Scale out. For more users, run several Ollama servers (each with its own GPU) behind a load balancer, a component that spreads incoming requests across them. Keep them identical: the same Ollama version, the same pinned model tags, and the same custom Modelfiles from version control. Treat a model update like a code release: test, then roll out.
Monitor. Log request counts, queue waits, errors, tokens per second and GPU memory (for example with nvidia-smi and the server logs). Check ollama ps for CPU spills after changes.
Know the limits. Ollama focuses on simplicity. For very high concurrency on large GPU servers, dedicated serving engines (such as vLLM or other high-throughput servers) usually serve more requests per GPU, at the cost of more setup. Many teams prototype on Ollama and move only their busiest models.
Security from the previous lesson applies: authentication and rate limits at a proxy in front of every server.
Example
# Simple load test: N concurrent chats, report latency and tokens/s
import concurrent.futures, time
from ollama import Client
client = Client(host="http://127.0.0.1:11434")
PROMPT = [{"role": "user", "content": "Write a 3-sentence product description for a desk lamp."}]
def one_call(_):
start = time.perf_counter()
r = client.chat(model="qwen3:8b", messages=PROMPT, options={"num_predict": 120})
return time.perf_counter() - start, r.eval_count / (r.eval_duration / 1e9)
for workers in (1, 4, 8):
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
results = list(pool.map(one_call, range(workers * 2)))
lat = sorted(r[0] for r in results)
tps = sum(r[1] for r in results) / len(results)
print(f"{workers} parallel: median {lat[len(lat)//2]:.1f}s, {tps:.0f} tok/s per request")
# 1 parallel: median 2.1s, 62 tok/s ... compare against OLLAMA_NUM_PARALLEL settingsRemember
- Size memory for weights + parallel slots × context; keep it all on the GPU
- Pin versions and model tags; scale out with identical servers behind a load balancer
- Monitor queueing and speed; consider dedicated engines for very high concurrency
Watch out for
- Letting every user pick any model on a shared GPU server, causing constant model reloads and long waits for everyone.