Running Ollama in Docker

Containers make Ollama easy to deploy on servers and alongside other services.

Docker runs programs in containers: isolated packages with everything the program needs. Ollama publishes an official image, ollama/ollama.

CPU only:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

-d runs it in the background.
-v ollama:/root/.ollama stores models in a Docker volume named ollama, so they survive when the container is replaced. Without it, every new container downloads models again.
-p 11434:11434 makes the API reachable on the host's port 11434.

NVIDIA GPUs: install the NVIDIA Container Toolkit on the host, configure Docker to use it, then add --gpus=all.

AMD GPUs: use the ollama/ollama:rocm image and pass the devices with --device /dev/kfd --device /dev/dri. The standard image also includes Vulkan support, used when the container can access GPU devices.

Run commands inside the container with docker exec:

docker exec -it ollama ollama run llama3.2

Docker Compose files work the same way: an ollama service with the volume, the port and GPU reservations. Other containers in the same Compose project reach it at http://ollama:11434, using the service name instead of localhost.

Pass settings as environment variables with -e, for example -e OLLAMA_KEEP_ALIVE=1h.

Publishing the port with -p 11434:11434 exposes the API on all of the host's network interfaces by default. On a shared network or server, bind it to localhost (-p 127.0.0.1:11434:11434) or protect it; the security lesson explains why.

Example

# NVIDIA GPU, persistent models, API only on this machine
docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  -e OLLAMA_KEEP_ALIVE=1h \
  --name ollama ollama/ollama

docker exec -it ollama ollama pull qwen3:8b
docker exec -it ollama ollama ps

# docker-compose.yml (excerpt)
# services:
#   ollama:
#     image: ollama/ollama
#     volumes: ["ollama:/root/.ollama"]
#     deploy:
#       resources:
#         reservations:
#           devices: [{driver: nvidia, count: all, capabilities: [gpu]}]
#   app:
#     environment: ["OLLAMA_HOST=http://ollama:11434"]   # service name, not localhost
# volumes: {ollama: {}}

Remember

ollama/ollama image; -v ollama:/root/.ollama keeps models
--gpus=all (NVIDIA) or the :rocm image with devices (AMD)
Other containers use the service name, e.g. http://ollama:11434

Watch out for

Forgetting the volume, so every container update re-downloads many gigabytes of models.

Check yourself

Why mount a volume at /root/.ollama when running the Ollama container?

Official docs: docs.ollama.com/docker

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks