TopolearnWorkers crash, deploys kill processes and brokers redeliver, so production tasks must be safe to run more than once and not lost.
A worker acknowledges a message to tell the broker it can be deleted. By default Celery acks just before executing the task (early ack): if the worker process dies mid-task, the task is lost. That's at-most-once.
With acks_late=True (per task) or task_acks_late=True, the ack happens after the task finishes. A crash before that means the broker redelivers the message to another worker: at-least-once. Add task_reject_on_worker_lost=True so a task whose process was killed (OOM, SIGKILL) is requeued too; beware that a task that always crashes its process then loops forever.
At-least-once delivery means duplicates are possible, so tasks must be idempotent: running twice has the same effect as once. Techniques:
- Use natural keys and upserts instead of blind inserts.
- Record a processed marker (unique task id or business key) in the same transaction as the work.
- Check state before acting ("already charged?") and use idempotency keys with external APIs.
With the Redis or SQS brokers, unacknowledged messages become visible again after the visibility timeout (Redis default: one hour, broker_transport_options={"visibility_timeout": ...}). Tasks running longer than it get redelivered while still running, so set it above your longest task or ETA.
Late acks also mean the prefetched messages of a worker are not lost on crash, making worker_prefetch_multiplier=1 a common companion for long tasks.
Example
from celery import Celery
app = Celery("proj", broker="redis://localhost:6379/0")
app.conf.update(
task_acks_late=True, # ack after the task finishes
task_reject_on_worker_lost=True, # requeue if the process is killed
worker_prefetch_multiplier=1, # take one message at a time
broker_transport_options={"visibility_timeout": 4 * 3600},
)
PROCESSED: set[str] = set() # stand-in for a DB table with a unique key
@app.task(bind=True)
def charge_order(self, order_id: int):
key = f"charge:{order_id}"
if key in PROCESSED: # idempotency check
return "already charged"
# payment_api.charge(order_id, idempotency_key=key)
PROCESSED.add(key) # in real code: same DB transaction
return "charged"Remember
- Default early ack = at-most-once; acks_late = at-least-once
- task_reject_on_worker_lost requeues tasks killed mid-run
- At-least-once means duplicates: make tasks idempotent
- Keep Redis/SQS visibility_timeout longer than tasks and ETAs
Watch out for
- Enabling acks_late on non-idempotent tasks like sending payments or emails without a dedupe key.