TopolearnSystems processing millions of tasks per day fail in ways small ones never show, mostly through broker and memory behavior.
ETA and countdown tasks are not held by the broker until due. The worker receives them immediately and keeps them in memory, unacknowledged, until their time comes. Thousands of far-future ETAs inflate worker memory, block prefetch slots, and on Redis or SQS get redelivered after the visibility timeout, creating duplicates. For long delays, store the job in your database with a run-at time and let a periodic task enqueue due jobs.
Priorities depend on the broker. RabbitMQ supports real per-queue priorities (x-max-priority on the queue plus priority= on calls). Redis emulates them with several lists, and prefetching dilutes the effect. Separate queues with dedicated workers are more predictable than priorities everywhere.
Payload size: messages and results live in broker and backend memory. Pass references (ids, object-store keys), not megabytes of data.
Backpressure: Celery won't stop producers when consumers fall behind. Monitor queue depth, set expires on tasks that lose value with time, use rate limits (rate_limit="100/m", enforced per worker) for fragile downstream APIs, and consider autoscaling workers on queue length.
Result backend load grows with every stored result; disable results where unused and expire the rest.
Finally, broker choice matters: RabbitMQ gives the strongest delivery semantics and features; Redis is simpler but trades durability and priority fidelity.
Example
from datetime import datetime, timedelta, timezone
from celery import Celery
app = Celery("proj", broker="redis://localhost:6379/0")
@app.task(rate_limit="100/m", ignore_result=True)
def call_partner_api(item_id): ...
@app.task(ignore_result=True)
def enqueue_due_jobs():
# runs every minute via beat: query rows with run_at <= now
due = [101, 102] # stand-in for a DB query
for job_id in due:
call_partner_api.apply_async(args=(job_id,), expires=300)
# Anti-pattern for long delays: the worker holds this in memory for 3 days
# and Redis redelivers it every visibility_timeout.
# call_partner_api.apply_async(args=(7,), eta=datetime.now(timezone.utc) + timedelta(days=3))
app.conf.beat_schedule = {"due-jobs": {"task": enqueue_due_jobs.name, "schedule": 60.0}}Remember
- ETA/countdown tasks sit unacked in worker memory until due
- For long delays, store run-at times in a database and enqueue when due
- Priorities are broker-specific; separate queues are more predictable
- Pass references, not big payloads; watch queue depth and use expires
Watch out for
- Scheduling password-expiry reminders 30 days out with eta on a Redis broker.