Make Celery Tasks Idempotent and Safe to Retry

Problem statement

In production, Celery tasks run more than once. Workers are killed mid-task during deploys and the task is redelivered; the Redis broker redelivers messages whose visibility timeout expires; network errors trigger retries after the side effect already happened; and users double-click buttons that enqueue work. Each duplicate can send a second email, charge a card twice, or create duplicate rows.

You cannot make delivery exactly-once, but you can make tasks idempotent: running them again produces the same result as running them once. This guide covers the patterns that achieve that in a Django app, and how to configure retries so transient failures recover without hammering dependencies.

1. Enqueue tasks with transaction.on_commit() so they never run before their data is committed.
2. Pass IDs, not objects, and reload state inside the task.
3. Record completion in the database (a status field or a unique "done" row) and check it at the start of the task.
4. Use database constraints (UniqueConstraint, get_or_create) or select_for_update() so concurrent duplicates cannot both proceed.
5. Send idempotency keys to external APIs that support them (payments, messaging).
6. Retry only transient errors, with autoretry_for, exponential backoff, jitter, and a maximum number of retries.

Step-by-step solution

1. Enqueue after commit

from functools import partial

from django.db import transaction

from billing.tasks import send_receipt


def complete_order(order):
    with transaction.atomic():
        order.status = "paid"
        order.save(update_fields=["status"])
        transaction.on_commit(partial(send_receipt.delay, order.pk))

Without on_commit, a fast worker can pick up the task before the transaction commits, find the old state, and do the wrong thing (or fail with DoesNotExist). If the transaction rolls back, the task is never sent.

2. Make re-runs detect finished work

@app.task(bind=True)
def send_receipt(self, order_id):
    order = Order.objects.get(pk=order_id)
    if order.receipt_sent_at:
        return "already sent"

    send_receipt_email(order)

    Order.objects.filter(pk=order_id, receipt_sent_at__isnull=True).update(receipt_sent_at=timezone.now())

This covers the common case: a task redelivered after it finished runs, sees the marker, and returns. A small window remains (crash after sending, before recording), which is acceptable for emails but not for payments; the next patterns close it.

3. Claim work atomically

When two copies of a task can run at the same time, a check followed by an action is a race. Let the database decide who wins:

from django.db import IntegrityError, transaction


class TaskRun(models.Model):
    key = models.CharField(max_length=200, unique=True)
    created = models.DateTimeField(auto_now_add=True)


@app.task
def generate_invoice(order_id):
    try:
        with transaction.atomic():
            TaskRun.objects.create(key=f"invoice:{order_id}")
            invoice = Invoice.objects.create(order_id=order_id, ...)
    except IntegrityError:
        return "duplicate, skipped"
    return invoice.pk

The unique key and the invoice are created in one transaction: either both exist or neither does, and a concurrent duplicate fails on the unique constraint. For state machines, select_for_update() serialises updates:

with transaction.atomic():
    order = Order.objects.select_for_update().get(pk=order_id)
    if order.status != "pending_fulfilment":
        return "not pending"
    order.status = "fulfilling"
    order.save(update_fields=["status"])

Keep these transactions short and never wrap slow external calls in them.

4. Use idempotency keys with external APIs

A payment request that times out may still have succeeded. Retrying without protection charges twice. Many APIs accept an idempotency key; Stripe, for example, takes an Idempotency-Key header and returns the original result for repeated keys:

@app.task(bind=True, autoretry_for=(stripe.APIConnectionError,), retry_backoff=True, max_retries=5)
def capture_payment(self, payment_id):
    payment = Payment.objects.get(pk=payment_id)
    if payment.captured:
        return "already captured"
    result = stripe.PaymentIntent.capture(
        payment.intent_id,
        idempotency_key=f"capture-{payment.pk}",
    )
    Payment.objects.filter(pk=payment.pk).update(captured=True, provider_status=result.status)

Derive the key from your own stable identifiers, never from random values generated per attempt. For APIs without idempotency support, record the request before sending it and reconcile unknown outcomes by querying the provider.

5. Configure retries

@app.task(
    bind=True,
    autoretry_for=(requests.ConnectionError, requests.Timeout),
    retry_backoff=True,        # 1s, 2s, 4s, 8s ...
    retry_backoff_max=600,     # cap the delay at 10 minutes
    retry_jitter=True,         # randomise delays to avoid thundering herds
    max_retries=6,
)
def sync_customer(self, customer_id):
    customer = Customer.objects.get(pk=customer_id)
    response = crm_session().put(f"{CRM_URL}/customers/{customer.external_id}", json=customer.as_crm(), timeout=10)
    response.raise_for_status()

• Retry only transient errors (connection failures, timeouts, 429, 503). Validation errors and 404s will not succeed on retry; let them fail and alert.
• Always set a timeout on network calls, or a hung connection holds the worker indefinitely.
• After max_retries, the task fails; make sure failures reach your error tracker.

For HTTP errors, raise a custom exception for retryable status codes and list it in autoretry_for.

6. Understand redelivery settings

CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 3600}

• acks_late acknowledges messages after the task finishes, so a worker crash causes redelivery instead of loss. That is only safe for idempotent tasks.
• With Redis as the broker, unacknowledged messages are redelivered after visibility_timeout (one hour by default). Tasks that run longer, or ETA/countdown tasks scheduled further ahead than the timeout, can be delivered twice. Keep tasks shorter than the timeout, or raise it above your longest task and ETA.
• prefetch_multiplier = 1 with acks_late avoids a worker holding many unacknowledged messages that are redelivered together if it dies.

Verification check:

Test idempotency directly: call each important task twice in a test (and concurrently if possible) and assert one side effect.

def test_generate_invoice_is_idempotent(db, order):
    generate_invoice(order.pk)
    generate_invoice(order.pk)
    assert Invoice.objects.filter(order=order).count() == 1

Explanation

Celery guarantees at-least-once delivery when configured with late acknowledgement, and at-most-once otherwise; neither is exactly-once. Networks add another source of ambiguity: a request that timed out may have succeeded. Idempotency moves the guarantee into your data: the database records what has been done, constraints stop concurrent duplicates, and external systems deduplicate with keys. With that in place, retries and redeliveries become safe, and you can configure Celery for reliability rather than hoping duplicates do not happen.

Edge cases / notes

Chords, chains, and groups

Workflows multiply the effect of duplicates. Make each step idempotent individually rather than relying on the workflow to run once.

Cleaning up task records

Tables of claimed keys grow. Delete rows older than your maximum retry and redelivery window with a periodic task.

When to automate this

Add a test helper that runs any task twice and compares side effects, and require it for tasks that touch money, email, or external systems. Monitor retry counts per task; a steady increase points to a failing dependency before tasks start exhausting their retries.

Next steps

• Set up the broker and workers with Configure Redis and Celery for Django Production.
• Tune prefetch and concurrency with Tune Celery Worker Concurrency and Prefetch for Django.
• Handle killed workers with Fix Celery WorkerLostError in Django.
• Review the rest of the worker setup with the Celery Production Checklist for Django.

FAQ

Why do Celery tasks run twice?

Redelivery after worker crashes or visibility timeouts, retries after partial success, and duplicate enqueues from users all cause repeats. Delivery is at-least-once at best.

How do I make a Celery task idempotent?

Record completion in the database, claim work with unique constraints or row locks, and use idempotency keys for external APIs.

Should I use acks_late in Celery?

Yes for idempotent tasks, so crashed tasks are redelivered rather than lost. Avoid it for tasks that are unsafe to repeat.

Which errors should Celery retry automatically?

Transient ones: connection errors, timeouts, rate limits, and temporary server errors. Do not retry validation errors or missing resources.