Fix Celery Tasks Executing Twice (Redis Visibility Timeout)

Problem statement

Customers receive the same email twice, an invoice is charged twice, or a report appears three times. The worker log shows one task ID received more than once, sometimes exactly an hour apart, or a burst of duplicates right after a deploy. Beat only scheduled it once and the code only called delay() once.

With Redis as the broker, Celery emulates acknowledgements using a visibility timeout. A message that a worker has taken but not acknowledged within that window is put back on the queue for another worker. Long-running tasks, tasks scheduled far in the future with eta or countdown, and workers killed during deploys all trigger redelivery.

1. Find the duplicate pattern: same task ID received twice means redelivery, different IDs mean it was sent twice.
2. Raise visibility_timeout in CELERY_BROKER_TRANSPORT_OPTIONS above your longest task runtime and longest eta/countdown.
3. Do not schedule tasks days ahead with eta; store the due time in the database and let Beat pick it up.
4. Stop workers gracefully during deploys so in-flight tasks finish or are requeued once.
5. Make side-effecting tasks idempotent with a unique key or a database flag.

Step-by-step solution

1. Tell redelivery apart from double sending

Search the worker logs for the task ID:

journalctl -u celery-worker --since "-1d" | grep "a1b2c3d4-..."

• Same ID received twice: the broker redelivered the message. Continue with this guide.
• Different IDs: your code or Beat sent it twice. See Fix Celery Beat Duplicate Tasks and check for delay() calls inside signals that fire more than once, or in transactions that are retried.

2. Check the visibility timeout against runtimes

The default is 3,600 seconds. Any task that is still running, or still waiting for its ETA, after one hour becomes visible again:

CELERY_BROKER_TRANSPORT_OPTIONS = {
    "visibility_timeout": 43200,   # 12 hours
}

Set it above the longest task runtime plus the longest ETA you use, with margin. The trade-off: if a worker really dies, its messages wait this long before redelivery. If you also use Redis for results, set CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS consistently.

3. Stop using far-future ETA and countdown

send_reminder.apply_async(args=[user.id], countdown=3 * 24 * 3600)   # avoid

A worker prefetches ETA tasks and holds them unacknowledged until they are due. Every visibility timeout that passes in between restores the message, and every worker that picks it up holds another copy. Store the due date instead:

Reminder.objects.create(user=user, due_at=timezone.now() + timedelta(days=3))

A periodic task every few minutes sends due reminders and marks them sent. This is also restart-safe and easy to cancel.

4. Shut workers down gracefully

When a worker receives SIGTERM, it stops taking new tasks and waits for running ones (warm shutdown). If your process manager kills it with SIGKILL first, tasks with acks_late=True are redelivered, and tasks that were half done run again from the start:

• systemd: KillSignal=SIGTERM and TimeoutStopSec= longer than your longest task.
• Docker: stop_grace_period: 10m in Compose.
• Kubernetes: terminationGracePeriodSeconds likewise.

5. Make side effects idempotent

Even with correct settings, at-least-once delivery means duplicates can happen. Guard anything with external effects:

@shared_task(bind=True, acks_late=True)
def charge_invoice(self, invoice_id):
    with transaction.atomic():
        inv = Invoice.objects.select_for_update().get(pk=invoice_id)
        if inv.charged_at:
            return "already charged"
        payment_provider.charge(inv, idempotency_key=f"invoice-{inv.pk}")
        inv.charged_at = timezone.now()
        inv.save(update_fields=["charged_at"])

The row lock plus the flag stops concurrent duplicates, and the provider's idempotency key protects against a crash between charging and saving.

Verification check: count executions per business key for a few days, for example by logging the invoice ID in the task and grouping the logs. After the changes, each key should appear once except for intentional retries.

Explanation

RabbitMQ tracks unacknowledged messages per connection and redelivers them only when that connection closes. Redis has no such mechanism, so kombu stores reserved messages in a sorted set with a timestamp and restores any that are older than the visibility timeout. This works well for short tasks and badly for anything long or scheduled far ahead. Celery guarantees at-least-once delivery on either broker, never exactly-once, so the application must tolerate repeats.

Edge cases / notes

• acks_late=True acknowledges after the task finishes, which is safer against crashes but increases redeliveries when a worker is killed. Combine it with idempotency, never alone.
• SQS has its own visibility timeout per queue; the same reasoning applies and the setting lives in the queue definition.
• Multiple Beat instances produce different task IDs, not the same ID twice. That is a separate fix.
• worker_prefetch_multiplier higher than 1 means a worker holds extra reserved messages that can be restored if it is slow. For long tasks use 1.

Next steps

• Build idempotent tasks with Make Celery Tasks Idempotent and Safe to Retry.
• Set prefetch and concurrency with Tune Celery Workers for Django.

FAQ

Why exactly one hour apart?

That is the default Redis visibility_timeout. A task still unacknowledged after an hour is restored to the queue.

Does RabbitMQ fix duplicate tasks?

It removes the visibility-timeout redelivery, but a killed worker still causes redelivery with acks_late. Idempotency is still needed.

Can I set visibility_timeout very high?

Yes, but messages from a worker that crashed are then delayed by that much before another worker gets them.

Is countdown safe for short delays?

Countdowns of seconds or minutes are fine. Days-long delays belong in the database.