TopolearnQueues fail quietly: without monitoring you learn about a stuck worker or growing backlog from angry users.
Celery has built-in remote control over the broker:
celery -A proj statuslists live workers.celery -A proj inspect activeshows running tasks;inspect reservedprefetched ones;inspect scheduledETA tasks;inspect registeredthe task names each worker knows (great for "task never runs" bugs).celery -A proj control rate_limit proj.tasks.send_email 10/morcontrol shutdownchange behavior at runtime.
Events: start workers with -E (or worker_send_task_events=True) and they publish task-received/started/succeeded/failed events. Flower (pip install flower, celery -A proj flower, port 5555) consumes them for a web dashboard with task history, worker status and a Prometheus metrics endpoint.
What to alert on: queue length (read directly from the broker, e.g. LLEN on the Redis list named after the queue), task failure rate, task runtime percentiles and the age of the oldest message.
Signals let your code hook into the lifecycle without touching tasks: task_prerun and task_postrun for timing and context (request ids for logs), task_failure to report to an error tracker, worker_process_init to set up per-process resources such as a database engine after fork.
Example
import time
from celery import Celery
from celery.signals import task_prerun, task_postrun, task_failure
app = Celery("proj", broker="redis://localhost:6379/0")
_started: dict[str, float] = {}
@task_prerun.connect
def on_start(task_id=None, task=None, **kw):
_started[task_id] = time.monotonic()
@task_postrun.connect
def on_end(task_id=None, task=None, state=None, **kw):
took = time.monotonic() - _started.pop(task_id, time.monotonic())
print(f"{task.name} {state} in {took:.2f}s") # or emit a metric
@task_failure.connect
def on_failure(task_id=None, exception=None, sender=None, **kw):
print(f"{sender.name} failed: {exception!r}") # send to error tracker
# $ celery -A proj inspect active
# $ celery -A proj flower --port=5555Remember
- inspect active/reserved/scheduled/registered for live introspection
- Enable events (-E) and use Flower for dashboards
- Alert on queue length, failure rate and runtime
- Signals hook into task and worker lifecycles
Watch out for
- Monitoring only worker liveness and missing a queue that grows for hours.