Tune Celery Worker Concurrency and Prefetch for Django

Problem statement

Celery workers started with default options often perform poorly once real workloads arrive:

• short tasks wait behind long ones because each worker process reserved several messages in advance
• CPU-heavy tasks run with too many processes and thrash the machine
• I/O-heavy tasks (HTTP calls, email) run with too few processes and leave the CPU idle
• a stuck task holds a worker process forever
• memory grows until the OOM killer terminates workers mid-task
• one burst of low-priority tasks delays password reset emails

Celery exposes the controls for all of these, but the defaults target general use rather than any specific workload. This guide explains the main settings, how they interact, and how to choose values based on measurements.

Quick answer

1. Separate workloads into queues (for example default, emails, reports) and run dedicated workers per queue.
2. For CPU-bound tasks, use the prefork pool with concurrency near the number of CPU cores.
3. For I/O-bound tasks, use higher prefork concurrency or a threads or gevent pool, after checking library compatibility.
4. Set worker_prefetch_multiplier = 1 for long or uneven tasks, often together with task_acks_late = True.
5. Set task_soft_time_limit and task_time_limit so stuck tasks cannot hold workers forever.
6. Use worker_max_tasks_per_child and worker_max_memory_per_child to recycle leaking processes.
7. Measure queue latency, task duration, and throughput before and after each change.

Step-by-step solution

1. Measure first

Useful numbers per task type:

• task duration (p50, p95): from logs, Flower, or task metrics
• queue wait time: time between enqueue and start
• throughput: tasks per minute per queue
• resource use: CPU and memory per worker host

A simple way to capture wait time is to record the enqueue time in task headers or arguments and log the difference when the task starts. Without these numbers, tuning is guesswork.

2. Route tasks to dedicated queues

# settings.py
CELERY_TASK_DEFAULT_QUEUE = "default"
CELERY_TASK_ROUTES = {
    "accounts.tasks.send_*": {"queue": "emails"},
    "reports.tasks.*": {"queue": "reports"},
}

Run workers per queue with settings suited to each:

celery -A config worker -Q emails  -n emails@%h  --concurrency 8 --prefetch-multiplier 4
celery -A config worker -Q reports -n reports@%h --concurrency 2 --prefetch-multiplier 1
celery -A config worker -Q default -n default@%h --concurrency 4

• Short, frequent tasks (emails) benefit from more concurrency and some prefetching.
• Long, heavy tasks (reports) use low concurrency and no prefetching, so one slow report does not block others that could start on another process.
• A burst in reports no longer delays emails.

-n gives each worker a unique node name, which matters when several workers run on one host.

3. Choose the pool and concurrency

For prefork, --concurrency defaults to the number of CPUs. Guidelines:

• CPU-bound (image processing, PDF generation, data crunching): concurrency ≈ CPU cores.
• I/O-bound (API calls, email, webhooks): concurrency can be several times the core count, limited by memory per process and by rate limits of the services you call.
• Database-heavy: each process may hold a database connection; keep total connections within limits.

With the threads pool, Django's database connections are per thread, so concurrency still determines connection count.

4. Understand prefetching

Each worker process reserves messages from the broker ahead of time: up to concurrency × worker_prefetch_multiplier messages per worker node (default multiplier 4). Reserved messages wait on that worker even if another worker is idle.

This is efficient for many short tasks but harmful for long ones. Example: a worker with concurrency 2 and multiplier 4 reserves 8 report tasks; two start, six wait, while a second idle worker receives nothing.

For long or variable tasks:

CELERY_WORKER_PREFETCH_MULTIPLIER = 1

or per worker with --prefetch-multiplier 1.

5. Late acknowledgement

By default, a message is acknowledged when a worker receives it, before execution. If the worker crashes, the task is lost. With late acknowledgement, it is acknowledged after completion:

CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True

• acks_late makes tasks survive worker crashes by redelivery, and combined with prefetch_multiplier = 1 limits reservations to what the worker is executing.
• reject_on_worker_lost requeues a task when the worker process executing it is killed (for example by the OOM killer), instead of acknowledging it.

The trade-off: tasks may run more than once. Only enable late acknowledgement for idempotent tasks, and consider the Redis visibility timeout for long tasks.

6. Time limits

A task stuck on a hung network call holds a worker process forever. Set limits:

CELERY_TASK_SOFT_TIME_LIMIT = 300
CELERY_TASK_TIME_LIMIT = 330

Or per task:

from celery import shared_task
from celery.exceptions import SoftTimeLimitExceeded


@shared_task(soft_time_limit=120, time_limit=150)
def build_report(report_id):
    try:
        generate(report_id)
    except SoftTimeLimitExceeded:
        mark_report_failed(report_id, reason="timed out")
        raise

• The soft limit raises SoftTimeLimitExceeded inside the task, allowing cleanup.
• The hard limit kills the process if the task does not stop.

Time limits are enforced by the prefork pool; they are not supported in the same way by all pools. Also set timeouts on network calls inside tasks, which is more precise than task-level limits.

7. Memory and leaks

Long-running worker processes can grow in memory due to fragmentation or leaks in libraries. Recycle them:

CELERY_WORKER_MAX_TASKS_PER_CHILD = 500
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 400_000  # kilobytes

• max_tasks_per_child replaces a child process after it has run that many tasks.
• max_memory_per_child replaces a child after a task leaves it above that resident memory size (in kilobytes).

Both apply to the prefork pool. Recycling costs some startup time, so do not set the task count very low without evidence.

8. Rate limits and external services

Protect third-party APIs and your own quotas:

@shared_task(rate_limit="30/m")
def sync_customer_to_crm(customer_id):
    ...

Celery's rate limits apply per worker instance, not globally. With several workers, the effective limit multiplies. For strict global limits, use a shared token bucket in Redis or a single dedicated worker for that queue.

9. Connections and Django

Worker processes use Django's ORM outside the request cycle. Close stale connections around tasks to avoid using broken connections after database restarts:

from celery.signals import task_postrun, task_prerun
from django.db import close_old_connections


@task_prerun.connect
def _close_before(**kwargs):
    close_old_connections()


@task_postrun.connect
def _close_after(**kwargs):
    close_old_connections()

10. Example configuration

CELERY_BROKER_URL = os.environ["CELERY_BROKER_URL"]
CELERY_TASK_DEFAULT_QUEUE = "default"
CELERY_TASK_ROUTES = {
    "accounts.tasks.send_*": {"queue": "emails"},
    "reports.tasks.*": {"queue": "reports"},
}
CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_SOFT_TIME_LIMIT = 300
CELERY_TASK_TIME_LIMIT = 330
CELERY_WORKER_MAX_TASKS_PER_CHILD = 1000
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 60 * 60 * 2}

Adjust per queue with worker command-line options where workloads differ.

11. Verify

celery -A config inspect active_queues
celery -A config inspect stats | grep -E '"pool"|"max-concurrency"|"prefetch_count"'
celery -A config inspect reserved

inspect reserved shows messages each worker has reserved but not started; with prefetch_multiplier = 1 and acks_late, it stays small.

Verification check:

• queue wait time for short tasks stays low during bursts of long tasks
• CPU utilization on worker hosts is high but not saturated for CPU-bound queues
• no tasks run longer than their hard limit
• worker memory stays within budget over days
• database connections from workers stay within limits

Explanation

Celery's throughput depends on matching three things to the workload: how many tasks run at once (pool and concurrency), how many messages each worker holds in reserve (prefetch), and when messages are considered done (acknowledgement). Separating queues lets you tune those settings per workload instead of compromising for all of them, while time and memory limits keep individual bad tasks from degrading the whole system.

Edge cases / notes

Autoscaling concurrency

--autoscale=max,min adjusts process count based on load. It adds churn and makes memory usage less predictable; fixed concurrency with more worker hosts is often easier to operate.

Task priorities

Priorities are broker-dependent and limited. Separate queues with dedicated workers are more reliable than priority settings for keeping urgent tasks fast.

Long-running tasks and deploys

Stopping workers during deploys sends SIGTERM; Celery's warm shutdown waits for running tasks. Give systemd or your orchestrator a stop timeout longer than typical task duration, or design tasks to resume.

Common mistakes

• One queue and one worker configuration for emails, reports, and imports.
• Default prefetching with long tasks, leaving idle workers while tasks wait.
• acks_late on non-idempotent tasks.
• No time limits, so hung HTTP calls hold processes indefinitely.

Next steps

• Review the whole setup with the Celery Production Checklist for Django.
• Debug tasks that never start with Fix Celery Tasks Stuck in PENDING.
• Manage worker services with Run Celery Workers with systemd for Django.
• Diagnose memory kills with Fix Gunicorn Workers Killed by the OOM Killer.

FAQ

What Celery concurrency should I use?

For CPU-bound tasks, about one process per CPU core. For I/O-bound tasks, more processes or a threads or gevent pool, limited by memory, database connections, and external rate limits.

What does worker_prefetch_multiplier do?

It controls how many messages each worker process reserves in advance. Lower values (such as 1) prevent long tasks from blocking reserved tasks while other workers are idle.

Should I enable task_acks_late in Celery?

For idempotent tasks where losing work on a crash is worse than running twice, yes. Pair it with prefetch_multiplier = 1 and a visibility timeout longer than your longest task.

How do I stop Celery tasks from running forever?

Set soft_time_limit and time_limit globally or per task, and add timeouts to network calls inside tasks.