Route Celery Tasks to Separate Queues

Problem statement

With one Celery queue, every task waits in the same line. A burst of report exports or image processing jobs fills the queue, and password reset emails, webhooks, and payment confirmations wait behind them for minutes. Users notice; support tickets follow.

Separate queues let you give each class of work its own line and its own workers: fast, latency-sensitive tasks never wait behind slow batch jobs, and heavy tasks run with settings suited to them (lower concurrency, higher memory limits, longer timeouts). This guide sets up routing and per-queue workers for a Django project.

1. Group tasks by latency and resource needs: for example default, email (fast, urgent), reports (slow, heavy).
2. Route them in settings with CELERY_TASK_ROUTES by task name or pattern.
3. Set CELERY_TASK_DEFAULT_QUEUE so unrouted tasks land in a known queue.
4. Run one worker service per queue group with -Q, each with suitable concurrency, time limits, and memory limits.
5. Use CELERY_WORKER_PREFETCH_MULTIPLIER = 1 for queues with long tasks.
6. Monitor length and oldest-message age per queue, and alert on the urgent ones.

Step-by-step solution

1. Decide on queues

Group by what matters operationally, not by app:

Three or four queues cover most projects. Each additional queue adds a worker service to operate.

2. Route tasks

# settings.py
CELERY_TASK_DEFAULT_QUEUE = "default"
CELERY_TASK_ROUTES = {
    "accounts.tasks.send_password_reset": {"queue": "email"},
    "billing.tasks.send_receipt": {"queue": "email"},
    "reports.tasks.*": {"queue": "reports"},
    "media.tasks.*": {"queue": "media"},
}

Routes match task names; with the standard autodiscovery, a task's name is its module path plus function name (reports.tasks.export_orders). Glob patterns like reports.tasks.* route every task in the module. If you set explicit names (@app.task(name="reports.export_orders")), route by those.

Routing in settings keeps decisions in one place. You can also override per call:

export_orders.apply_async(args=[export.pk], queue="reports")

3. Run workers per queue

# /etc/systemd/system/celery-email.service
[Unit]
Description=Celery worker (email, default)
After=network.target

[Service]
User=myapp
WorkingDirectory=/srv/myapp/current
EnvironmentFile=/etc/myapp.env
ExecStart=/srv/myapp/current/.venv/bin/celery -A config worker \
  -Q email,default -n email@%%h --concurrency=8 --loglevel=INFO
Restart=always
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/celery-reports.service
[Service]
User=myapp
WorkingDirectory=/srv/myapp/current
EnvironmentFile=/etc/myapp.env
ExecStart=/srv/myapp/current/.venv/bin/celery -A config worker \
  -Q reports -n reports@%%h --concurrency=2 \
  --max-memory-per-child=800000 --prefetch-multiplier=1 \
  --soft-time-limit=1700 --time-limit=1800 --loglevel=INFO
Restart=always
TimeoutStopSec=1900

• -Q lists the queues a worker consumes. The email worker also takes default, which is fine when both are fast.
• -n name@%%h gives each worker a unique node name (%%h is the host name; the double % escapes systemd's own specifiers).
• The reports worker has low concurrency, a per-child memory cap, strict time limits, and a stop timeout longer than the time limit so deploys do not kill running exports.
• --prefetch-multiplier=1 stops a worker from reserving several long tasks while others sit idle.

Enable and start them:

sudo systemctl daemon-reload
sudo systemctl enable --now celery-email celery-reports
celery -A config inspect active_queues

4. Keep a worker for every queue

A queue without a consumer silently accumulates messages. If you add a route to a new queue, deploy a worker for it at the same time. Check that every routed queue has consumers:

celery -A config inspect active_queues --json | python3 -c "
import json, sys
queues = {q['name'] for w in json.load(sys.stdin).values() for q in w}
print(sorted(queues))
"

task_create_missing_queues is on by default, so Celery creates queues on first use; it does not start workers for them.

5. Rate limit where needed

Per-task rate limits protect external APIs:

@app.task(rate_limit="30/m")
def sync_to_crm(customer_id):
    ...

The limit applies per worker instance, not globally. For a global limit across many workers, route the task to a dedicated queue served by a single worker process, or implement a shared token bucket in Redis.

6. Monitor per queue

With the Redis broker, each queue is a list:

redis-cli -u "$CELERY_BROKER_URL" LLEN email
redis-cli -u "$CELERY_BROKER_URL" LLEN reports

Alert on the urgent queues' length and on how long the oldest message has waited (Flower and Prometheus exporters expose these). A growing email queue is an incident; a growing reports queue during business hours may be normal.

Verification check:

python manage.py shell -c "
from accounts.tasks import send_password_reset
r = send_password_reset.delay(1)
print(r.id)
"
celery -A config inspect active --json | head -c 500
journalctl -u celery-email -n 20 --no-pager | grep send_password_reset

The task should appear in the email worker's log within a second, even while the reports worker is busy.

Explanation

Celery routes each message to a named queue in the broker at publish time. Workers consume only the queues they are told to, so the mapping of tasks to queues and queues to workers determines which work can delay which. Separating by latency needs puts a hard boundary between batch work and interactive work, and per-queue worker settings let each class of task run under limits that fit it, rather than a compromise that fits none.

Edge cases / notes

Priorities instead of queues

Celery supports message priorities, but with Redis they are emulated with multiple lists and are less predictable than separate queues. Separate queues with dedicated workers are simpler to reason about.

Celery Beat

Periodic tasks follow the same routes. Heavy scheduled jobs therefore land on the heavy queue automatically when routed by module.

When to automate this

Keep routes in settings under version control, generate worker units from a list of queue definitions (name, concurrency, limits), and add a CI check that every queue named in CELERY_TASK_ROUTES has a worker definition.

Next steps

• Tune each worker with Tune Celery Worker Concurrency and Prefetch for Django.
• Supervise multiple worker services with Run Celery Workers with systemd for Django.
• Watch queues and workers with Monitor Celery with Flower.
• If tasks sit unprocessed, see Fix Celery Tasks Stuck in PENDING.

FAQ

How do I send a Celery task to a specific queue?

Add a rule to CELERY_TASK_ROUTES mapping the task name to a queue, or pass queue= to apply_async.

How do I make a Celery worker consume only one queue?

Start it with -Q queue_name. A worker can consume several queues with a comma-separated list.

Why are tasks piling up in a new Celery queue?

No worker consumes that queue. Start a worker with -Q including it.

Are Celery rate limits global?

No. rate_limit applies per worker instance. Use a single dedicated worker or a shared limiter for global limits.