Configuring a Celery app

Celery · Essentials · 10 min

Real projects keep settings in one place, and a few defaults (serializer, timezone, prefetch) are worth setting deliberately.

Celery settings are lowercase names like task_serializer or broker_url. Set them with app.conf.update(...), or keep them in a module and load it with app.config_from_object("celeryconfig"). With Django, the usual setup is app.config_from_object("django.conf:settings", namespace="CELERY"), so CELERY_BROKER_URL in settings.py maps to broker_url, plus app.autodiscover_tasks() to import each app's tasks.py.

Settings worth knowing early:

  • broker_url, result_backend.
  • task_serializer, result_serializer, accept_content: JSON by default. Pickle is disabled by default because unpickling messages can execute arbitrary code.
  • timezone and enable_utc (on by default) for ETAs and beat schedules.
  • result_expires for result cleanup.
  • task_acks_late, worker_prefetch_multiplier: delivery behavior, covered later.
  • task_default_queue (default celery).

A typical project layout is a package with celery.py defining the app and a tasks.py per area; the worker is started with celery -A proj worker, where proj is a package or module containing an attribute named app or celery.

Configuration must be loaded identically by clients and workers; mismatched serializers or task names are a frequent cause of tasks that "never run". celery -A proj report prints the effective configuration.

Example

# proj/celery.py
from celery import Celery

app = Celery("proj")
app.conf.update(
    broker_url="redis://localhost:6379/0",
    result_backend="redis://localhost:6379/1",
    task_serializer="json",
    accept_content=["json"],
    result_expires=3600,          # seconds
    timezone="Europe/Berlin",
    enable_utc=True,
    include=["proj.tasks"],       # modules the worker imports
)

# Django alternative:
# app.config_from_object("django.conf:settings", namespace="CELERY")
# app.autodiscover_tasks()

# $ celery -A proj worker -l INFO

Remember

  • Settings are lowercase names set via app.conf or config_from_object
  • JSON is the default and safest serializer
  • With Django, namespace='CELERY' maps CELERY_* settings
  • Clients and workers must share the same configuration and task modules

Watch out for

  • Enabling pickle to pass complex objects, opening a code-execution hole if the broker is reachable.

Check yourself

  1. With namespace='CELERY', which Django setting configures the broker?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.celeryq.dev/en/stable/userguide/configuration.html