Run Celery With RabbitMQ for Django in Production

Problem statement

Redis is the most common Celery broker for Django, but it emulates message acknowledgement with a visibility timeout, which causes duplicate deliveries for long or delayed tasks and offers limited durability. RabbitMQ is a purpose-built message broker: it tracks acknowledgements per connection, persists messages to disk, and supports replicated quorum queues. For task workloads where reliability matters more than simplicity (payments, order processing, long-running jobs), it is often the better broker. It also adds an operational component you must install, secure, and monitor.

1. Install RabbitMQ (packages or the official Docker image) and enable the management plugin.
2. Create a vhost and a dedicated user: rabbitmqctl add_vhost myapp, add_user, set_permissions.
3. Set CELERY_BROKER_URL = "amqp://myapp:<password>@rabbitmq:5672/myapp".
4. Use quorum queues for durability (x-queue-type: quorum) and CELERY_TASK_ACKS_LATE = True for idempotent tasks.
5. Set CELERY_WORKER_PREFETCH_MULTIPLIER = 1 for long tasks.
6. Keep a separate result backend (Redis or the database) if you need results; RabbitMQ is not a result store.

Step-by-step solution

1. Run RabbitMQ

With Docker Compose:

services:
  rabbitmq:
    image: rabbitmq:4-management
    hostname: rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS_FILE: /run/secrets/rabbitmq_admin
    volumes:
      - rabbitmq:/var/lib/rabbitmq
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
      interval: 10s
      retries: 6

Set a fixed hostname: RabbitMQ stores data under a node name derived from it, and a changing hostname makes it look like a new, empty node. On a VM, install from the RabbitMQ team's repositories so you get a current version and a matching Erlang.

2. Create a vhost and user for the app

rabbitmqctl add_vhost myapp
rabbitmqctl add_user myapp 'a-long-random-password'
rabbitmqctl set_permissions -p myapp myapp ".*" ".*" ".*"
rabbitmqctl delete_user guest

The default guest user only works from localhost, but delete it anyway. Each app gets its own vhost so queues and permissions are isolated.

3. Configure Celery

CELERY_BROKER_URL = env("CELERY_BROKER_URL")   # amqp://myapp:...@rabbitmq:5672/myapp
CELERY_RESULT_BACKEND = env("CELERY_RESULT_BACKEND", default=None)   # e.g. redis://redis:6379/1
CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True
CELERY_TASK_DEFAULT_QUEUE = "default"

Quorum queues replicate messages across cluster nodes and survive node failure. Declare them through kombu queues:

from kombu import Queue
CELERY_TASK_QUEUES = [
    Queue("default", queue_arguments={"x-queue-type": "quorum"}),
    Queue("reports", queue_arguments={"x-queue-type": "quorum"}),
]

Quorum queues do not support some classic features (such as global QoS); Celery 5.5+ detects quorum queues and adjusts. Check the Celery docs for your version.

4. Secure the broker

• Do not expose port 5672 or the management UI (15672) publicly; bind to the private network.
• Use TLS (amqps://, port 5671) when traffic crosses networks or hosts.
• Give the app user permissions only on its vhost.
• Set a vm_memory_high_watermark and disk free limit so RabbitMQ applies back-pressure rather than crashing.

5. Monitor queues

The management UI and the rabbitmq_prometheus plugin show queue depth, consumer count, message rates, and unacknowledged messages. Alert on growing queue depth with no consumers, and on unacked messages that never drop (stuck workers). Flower works with RabbitMQ too.

Verification check: celery -A config inspect ping returns pong, the management UI shows your queues with consumers attached, and a test task appears briefly in the queue and then is acknowledged. Restart RabbitMQ and confirm queued messages survive.

Explanation

AMQP brokers track every delivered message as unacknowledged until the consumer acks it. If the consumer's connection closes, unacked messages return to the queue immediately; otherwise they stay with that consumer indefinitely, however long the task takes. That removes the main cause of duplicate Celery tasks on Redis (the visibility timeout) and makes long tasks and ETAs safer. Durable, replicated queues add protection against broker failure, at the cost of running and monitoring another stateful service.

Edge cases / notes

• ETA and countdown tasks are held by workers unacknowledged; with very long delays they occupy worker memory. Store far-future jobs in the database instead.
• Consumer timeout. RabbitMQ closes channels whose deliveries stay unacked longer than consumer_timeout (30 minutes by default); raise it for very long tasks.
• Clustering needs at least three nodes for quorum queues to tolerate one failure.
• Managed options include Amazon MQ for RabbitMQ and CloudAMQP.

Next steps

• The Redis-based setup is in Configure Redis and Celery for Django in Production.
• Review worker settings with Tune Celery Workers for Django.

FAQ

Is RabbitMQ better than Redis for Celery?

For delivery guarantees and long tasks, yes. Redis is simpler to run and often already in the stack.

Can RabbitMQ store Celery results?

Use a separate result backend such as Redis or the database; the RPC backend has limits.

Do I still need idempotent tasks with RabbitMQ?

Yes. Delivery is still at least once, for example after a worker crash.

What are quorum queues?

Replicated, durable RabbitMQ queues based on Raft, recommended for important messages.