TopolearnThe right pool and prefetch settings decide throughput, memory use and how fairly long tasks are distributed.
A worker runs tasks in an execution pool, chosen with --pool:
prefork(default): child processes via multiprocessing. Best for CPU-bound work and isolation; a crash or leak affects one child.threads: a thread pool; light, good for IO-bound tasks that release the GIL.gevent/eventlet: green threads for very high concurrency IO work (hundreds of HTTP calls); requires monkey-patch-friendly libraries.solo: one task at a time in the main process; handy for debugging.
-c N / --concurrency sets the number of processes or threads (default: number of CPUs).
Prefetch: each worker reserves worker_prefetch_multiplier x concurrency messages ahead of time (multiplier default 4). Great for many short tasks, bad for long ones: a worker can sit on reserved messages while others are idle. For long tasks use a multiplier of 1, often with acks_late.
Long-running workers can leak memory. worker_max_tasks_per_child=1000 replaces a child after that many tasks, and worker_max_memory_per_child=200_000 (KiB) after a memory threshold.
Scale by adding worker processes or machines rather than one giant -c. Remember each prefork child holds its own database connections, so pools and pool sizes multiply.
Example
# CPU-heavy image processing: processes, one message at a time
# $ celery -A proj worker -Q images --pool=prefork -c 4 \
# --prefetch-multiplier=1 --max-tasks-per-child=200
# IO-heavy webhooks: many green threads
# $ celery -A proj worker -Q webhooks --pool=gevent -c 200
# Local debugging: one task at a time in the main process
# $ celery -A proj worker --pool=solo -l DEBUG
# Equivalent settings in config:
from celery import Celery
app = Celery("proj", broker="redis://localhost:6379/0")
app.conf.update(
worker_prefetch_multiplier=1,
worker_max_tasks_per_child=200,
worker_max_memory_per_child=300_000, # KiB, about 300 MB
)Remember
- prefork for CPU work, threads/gevent for IO-bound work, solo for debugging
- Prefetch = multiplier x concurrency messages reserved per worker
- Use prefetch 1 for long tasks to keep distribution fair
- Recycle children with max_tasks_per_child / max_memory_per_child
Watch out for
- Using gevent with libraries that block (e.g. a C database driver without patching), stalling the whole worker.