Retry Celery Tasks With Exponential Backoff in Django

Problem statement

Celery tasks that call email providers, payment APIs, or partner services fail intermittently: timeouts, 503 responses, connection resets. Without retries, work is lost. With naive retries (immediately, forever), a struggling service gets hammered by every worker at once, and broken tasks loop endlessly. Exponential backoff with jitter spaces retries out, gives dependencies time to recover, and stops after a sensible number of attempts.

1. Use autoretry_for=(ConnectionError, Timeout) with retry_backoff=True, retry_backoff_max=600, and retry_jitter=True.
2. Set max_retries (for example 5 to 8) so failures eventually surface.
3. Retry only transient errors; let bugs (ValueError, KeyError, 4xx responses) fail immediately.
4. Make tasks idempotent, because a retry may run after a partial success.
5. After the final failure, record it (Sentry, a failed-jobs table) for manual follow-up.

Step-by-step solution

1. Automatic retries with backoff

import requests
from celery import shared_task

TRANSIENT = (requests.ConnectionError, requests.Timeout)

@shared_task(
    bind=True,
    autoretry_for=TRANSIENT,
    retry_backoff=True,        # 1s, 2s, 4s, 8s ... (base 1 second)
    retry_backoff_max=600,     # cap a single wait at 10 minutes
    retry_jitter=True,         # randomize each wait between 0 and the computed value
    max_retries=8,
)
def push_invoice(self, invoice_id):
    invoice = Invoice.objects.get(pk=invoice_id)
    resp = requests.post(ERP_URL, json=invoice.as_payload(), timeout=10)
    if resp.status_code >= 500:
        resp.raise_for_status()       # HTTPError is not in TRANSIENT; handle below
    ...

retry_backoff=True uses a base of 1 second; pass an integer to change the base (for example retry_backoff=5 gives 5, 10, 20, 40...). Jitter prevents many tasks that failed together from retrying at the same instant.

2. Manual retries for status codes

HTTP errors need a decision per status. Use self.retry explicitly:

@shared_task(bind=True, max_retries=8)
def push_invoice(self, invoice_id):
    resp = requests.post(ERP_URL, json=payload(invoice_id), timeout=10)
    if resp.status_code in (429, 502, 503, 504):
        delay = int(resp.headers.get("Retry-After", min(600, 2 ** self.request.retries)))
        raise self.retry(countdown=delay)
    resp.raise_for_status()   # other 4xx: a bug or bad data, fail now

self.request.retries counts previous attempts, so you can compute your own backoff.

3. Choose what to retry

Retrying bugs wastes resources and delays the alert. If an object might not exist yet because the task ran before the transaction committed, fix the enqueueing (transaction.on_commit) rather than retrying.

4. Make retries safe

A retry can happen after the first attempt partly succeeded, for example the ERP received the invoice but the response timed out. Use idempotency keys when the API supports them, check state before acting, and record completion:

headers = {"Idempotency-Key": f"invoice-{invoice.pk}-v{invoice.version}"}

Combine retries with acks_late=True only when tasks are idempotent; then worker crashes also lead to re-execution.

5. Handle final failures

After max_retries, Celery raises MaxRetriesExceededError (or re-raises the original exception with autoretry). Capture it:

from celery.signals import task_failure

@task_failure.connect
def record_failure(sender=None, task_id=None, exception=None, args=None, **kw):
    FailedJob.objects.create(task=sender.name, task_id=task_id, args=args, error=repr(exception))

Sentry's Celery integration reports the final failure automatically. Provide an admin action or management command to re-run failed jobs after the root cause is fixed.

Explanation

When a task raises Retry (directly or via autoretry_for), Celery publishes a new message for the same task with an incremented retry count and an ETA in the future, then acknowledges the original. Exponential backoff increases that delay with each attempt, so a short outage costs a few quick retries while a long one does not flood the dependency. Jitter decorrelates retries from many tasks that failed together. The retry count limit turns persistent failures into visible errors.

Edge cases / notes

• ETA and brokers. Retries with countdown are held by workers until due; very long delays interact with Redis and SQS visibility timeouts, so cap waits at minutes, not hours.
• Database errors. Retrying after OperationalError should start a fresh transaction; do not retry inside an atomic block that already failed.
• Chains. A retried task in a chain delays the rest of the chain; set expectations accordingly.

Next steps

• Make tasks safe to repeat with Make Celery Tasks Idempotent With Retries.
• Pace calls to external APIs with Rate Limit Celery Tasks in Django.
• Review the rest with the Django Celery Production Checklist.

FAQ

What does retry_backoff=True do?

It makes automatic retries wait exponentially longer: 1, 2, 4, 8 seconds and so on, up to retry_backoff_max.

Should I retry all exceptions?

No. Retry transient errors only; let programming errors and permanent 4xx responses fail immediately.

What is retry_jitter for?

It randomizes the wait so many failed tasks do not retry at exactly the same time.

What happens after max_retries?

The task fails. Record the failure and alert so someone can fix the cause and re-run it.