TopolearnCelery is how Python apps move slow work (emails, reports, ML jobs) out of the request path, and every feature builds on this picture.
Celery is a distributed task queue. Your application (the client) doesn't run a slow function itself; it sends a message describing the call to a broker, a message queue such as Redis or RabbitMQ. One or more worker processes, possibly on other machines, consume messages and execute the function. Optionally a result backend (Redis, a database) stores return values so the client can fetch them later.
You create a Celery app with a name and a broker URL, then turn functions into tasks with @app.task. The task's name defaults to its module path plus function name (tasks.add), and that name, not the function object, is what travels in the message. So the worker must import the same code.
Start a worker from the shell with celery -A tasks worker --loglevel=INFO, where -A points to the module (or package.module:app) holding the app. The worker logs the tasks it registered and every task it receives.
Two rules to absorb now:
- Task arguments and results are serialized, as JSON by default. Pass ids and plain data, not ORM objects, open files or sockets.
- Tasks run in a different process, maybe much later. Don't rely on in-memory state from the caller.
Redis is the easiest broker to start with locally (docker run -p 6379:6379 redis); RabbitMQ is the most feature-complete.
Example
# tasks.py
from celery import Celery
app = Celery(
"tasks",
broker="redis://localhost:6379/0",
backend="redis://localhost:6379/1", # optional: store results
)
@app.task
def add(x, y):
return x + y
@app.task
def send_welcome_email(user_id: int) -> None:
# load the user by id inside the task, not in the caller
print(f"sending welcome email to user {user_id}")
# $ celery -A tasks worker --loglevel=INFO
# [tasks]
# . tasks.add
# . tasks.send_welcome_emailRemember
- Client sends messages to a broker; workers consume and run them
- @app.task registers a function under a name like module.func
- Start workers with celery -A <module> worker
- Arguments are serialized: pass ids and plain data
Watch out for
- Passing an ORM object or file handle as a task argument.