Use Amazon SQS as the Celery Broker for Django

Problem statement

Your Django app runs on AWS and you want Celery without operating Redis or RabbitMQ as a broker. Amazon SQS is fully managed, scales without capacity planning, and works with IAM roles instead of passwords. Celery supports SQS as a transport, but with differences from Redis: no built-in result backend, a visibility timeout that decides when unacknowledged tasks reappear, limited support for remote control and events, and per-request pricing.

1. Install the transport: pip install "celery[sqs]" (it pulls in boto3 and pycurl requirements depending on version).
2. Set CELERY_BROKER_URL = "sqs://" and let boto3 use the IAM role of the task, instance, or pod.
3. Set broker_transport_options with region, visibility_timeout longer than your longest task, polling_interval, and wait_time_seconds for long polling.
4. Create queues in advance (or let Celery create them) and use a queue_name_prefix per environment.
5. Use a separate result backend (Redis, the database, or none), and avoid long ETA or countdown tasks.

Step-by-step solution

1. Install and configure

CELERY_BROKER_URL = "sqs://"
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "region": "eu-central-1",
    "queue_name_prefix": "shop-prod-",
    "visibility_timeout": 3600,
    "polling_interval": 1,
    "wait_time_seconds": 20,
}
CELERY_TASK_DEFAULT_QUEUE = "default"
CELERY_RESULT_BACKEND = None
CELERY_WORKER_ENABLE_REMOTE_CONTROL = False

With an empty URL, Kombu uses boto3's default credential chain: ECS task role, EC2 instance profile, EKS pod identity, or environment variables. Avoid putting access keys in the URL.

2. Grant IAM permissions

The role used by web and workers needs:

{
  "Effect": "Allow",
  "Action": [
    "sqs:SendMessage", "sqs:ReceiveMessage", "sqs:DeleteMessage",
    "sqs:ChangeMessageVisibility", "sqs:GetQueueAttributes", "sqs:GetQueueUrl",
    "sqs:ListQueues", "sqs:CreateQueue"
  ],
  "Resource": "arn:aws:sqs:eu-central-1:123456789012:shop-prod-*"
}

Drop sqs:CreateQueue if you create queues with Terraform or CloudFormation and set predefined_queues in the transport options.

3. Choose the visibility timeout

When a worker receives a message, SQS hides it for the visibility timeout. If the worker does not delete it (acknowledge) in time, the message becomes visible again and another worker runs the task. Set visibility_timeout above your longest task's runtime, including retries in the same execution. With acks_late = True, acknowledgement happens after the task finishes, so a task longer than the timeout runs twice. See Fix Celery Task Executed Multiple Times.

4. Avoid long ETA tasks

Celery implements eta and countdown by holding messages in the worker's memory until they are due. With SQS, a held message must stay invisible the whole time, which conflicts with the visibility timeout and the 12-hour maximum. Keep countdowns short (minutes), and for tasks scheduled far in the future, store them in the database and enqueue them from a periodic job.

5. FIFO queues and dead letters

Standard queues provide at-least-once delivery without strict ordering, which suits Celery. FIFO queues work with extra settings but limit throughput and need MessageGroupId handling. Configure a dead-letter queue with a redrive policy so messages that keep failing are moved aside instead of looping forever.

Explanation

Kombu's SQS transport maps each Celery queue to an SQS queue and uses long polling to receive messages. Acknowledging a task deletes the message; failing to acknowledge in time lets SQS redeliver it. SQS has no pub/sub broadcast, so Celery features that use broadcast exchanges, such as remote control commands and events, are limited. In return, there is no broker to patch, scale, or secure with passwords, and messages are stored durably across availability zones.

Edge cases / notes

• Costs. Each receive, send, and delete is a billed request; long polling (wait_time_seconds = 20) reduces empty receives.
• Local development. Use Redis locally or LocalStack for SQS emulation; keep transport options environment-specific.
• Message size. SQS messages are limited in size (256 KB historically, larger with newer limits); pass IDs, not large payloads.

Next steps

• Compare with Redis in Configure Redis and Celery for Django in Production.
• Run workers on ECS with Deploy Django on AWS ECS Fargate.
• Review AWS settings with the Django AWS Production Checklist.

FAQ

Can SQS store Celery task results?

No. Use Redis, the Django database via django-celery-results, or no result backend.

Why do my tasks run twice with SQS?

The visibility timeout expired before the task finished. Raise it above your longest task or split the task.

Do I need access keys for SQS?

No. Use IAM roles for ECS tasks, EC2 instances, or EKS pods.

Does Flower work with SQS?

Only partly, because SQS does not support the broadcast messaging that events and remote control use.