Topolearn

Learn Celery

Distributed task queue for running Python work in background workers

16 lessons · 3 hours · 4 levels · covers Celery 5.6

Celery runs Python functions asynchronously in worker processes, fed by a message broker such as Redis or RabbitMQ. Web apps use it to move emails, reports, media processing and ML jobs out of the request path, to schedule periodic jobs and to build multi-step workflows. Learning it means learning reliable background processing: retries, idempotency, routing and monitoring.

Sign in free to track progressOfficial docs ↗

Before you start

Celery builds on these; the tree links to the exact lessons you need.

PythonPractical level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics4 lessons · 42 min in total
2. PracticalUse it productively at work, with the usual idioms9 lessons · 2 hours in total
3. AdvancedThe less common features, performance and trade-offs13 lessons · 2 hours in total
4. ExpertInternals and the edge cases experts know16 lessons · 3 hours in total

The Celery tree

CeleryBrokers, workers and your first taskBrokers, workers andyour first taskCalling tasks: delay and apply_asyncCalling tasks: delayand apply_asyncResults and task statesResults and taskstatesConfiguring a Celery appConfiguring a CeleryappRetries and error handlingRetries and errorhandlingCanvas: chains, groups and chordsCanvas: chains,groups and chordsPeriodic tasks with beatPeriodic tasks withbeatQueues and routingQueues and routingTesting Celery tasksTesting Celery tasksDefining functionsDefining functionsModules and importsModules and importsExceptions in depthExceptions in depthTesting with pytest and unittestTesting with pytestand unittestThreads, processes and concurrent.futuresThreads, processesand concurrent.fut…
Essentials and Practical shown (9 of 16 lessons); the full list is below. Click a lesson to read it.

All 16 lessons, in order

Getting started

  1. Brokers, workers and your first task · Essentials · 12 min

    Understand Celery's moving parts (client, broker, worker, result backend), define a task and start a worker.

  2. Calling tasks: delay and apply_async · Essentials · 10 min

    Queue tasks with delay(), and use apply_async() for countdowns, ETAs, expiry and queue selection.

  3. Results and task states · Essentials · 10 min

    Use AsyncResult to check state, fetch return values with get(), and decide when to ignore results altogether.

  4. Configuring a Celery app · Essentials · 10 min

    Configure Celery with app.conf or a config module: serializers, timezone, result expiry and framework integration.

Writing tasks

  1. Retries and error handling · Practical · 12 min

    Retry failed tasks with self.retry() or autoretry_for, using exponential backoff, jitter and max_retries.

  2. Testing Celery tasks · Practical · 10 min

    Unit-test task logic directly, use eager mode or apply() for integration-style tests, and know the pytest helpers.

Workflows, scheduling and routing

  1. Canvas: chains, groups and chords · Practical · 14 min

    Compose tasks into workflows with signatures, chain, group and chord instead of waiting on results.

  2. Periodic tasks with beat · Practical · 10 min

    Schedule recurring tasks with celery beat using intervals and crontab expressions, and run exactly one scheduler.

  3. Queues and routing · Practical · 10 min

    Send tasks to named queues with task_routes or queue=, and dedicate workers to queues with -Q.

Running in production

  1. Delivery guarantees and idempotent tasks · Advanced · 14 min

    Understand acknowledgement timing, acks_late, visibility timeouts and why tasks must be idempotent.

  2. Worker pools, concurrency and prefetch · Advanced · 12 min

    Choose between prefork, threads, gevent and solo pools, size concurrency, and tune prefetch and child recycling.

  3. Time limits and revoking tasks · Advanced · 10 min

    Stop runaway tasks with soft and hard time limits, clean up on SoftTimeLimitExceeded, and revoke tasks.

  4. Monitoring, inspect and signals · Advanced · 10 min

    Watch workers with celery inspect and control, Flower and events, and hook into task lifecycle signals.

Internals and scale

  1. Task messages and the protocol · Expert · 14 min

    What a Celery task message contains (protocol v2 headers and body), how Kombu transports it, and what that means for compatibility.

  2. How groups and chords really work · Expert · 12 min

    How chords track completion in the result backend, why they need results stored, and their failure and performance characteristics.

  3. At scale: ETAs, priorities and backpressure · Expert · 14 min

    Avoid the scaling traps: ETA tasks held in worker memory, broker-specific priorities, oversized payloads and unbounded queues.