Fix Celery WorkerLostError in Django

Problem statement

Some Celery tasks fail with:

celery.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) Job: 1423.

or signal 11 (SIGSEGV), or exitcode 1. The task did not raise an exception in your code; the child process running it died. With the default prefork pool, the Celery main process notices the child is gone, marks the task as failed with WorkerLostError, and starts a replacement child.

Signal 9 means something killed the process forcibly. On Django servers, that is almost always the kernel's out-of-memory killer, a container or systemd memory limit, or a process manager that ran out of patience during a restart. This guide finds which one and fixes it.

1. Note the signal: 9 (SIGKILL) points at memory limits or forced stops, 11 (SIGSEGV) at a crash in a C extension.
2. Check the kernel log for OOM kills: journalctl -k | grep -i -E "oom|killed process".
3. Check container and systemd limits (docker inspect, MemoryMax=), and whether the kills coincide with deploys.
4. Reduce memory per task (chunk querysets, stream files), cap per-child memory with --max-memory-per-child, and lower concurrency if needed.
5. Give workers enough stop time during deploys so running tasks finish instead of being killed.
6. Decide deliberately whether lost tasks are redelivered (acks_late with task_reject_on_worker_lost), and make those tasks idempotent.

Step-by-step solution

1. Match the error to a cause

sudo journalctl -k --since "2 hours ago" | grep -i -E "out of memory|oom|killed process"
sudo journalctl -u celery-worker --since "2 hours ago" | grep -E "WorkerLostError|signal"

A line like Out of memory: Killed process 81234 (celery) total-vm:... anon-rss:1843200kB at the same time as the task failure confirms the OOM killer. In containers, docker inspect --format '{{.State.OOMKilled}}' <container> shows OOM kills of the whole container; child kills inside a container show up in the host's kernel log as well.

If the failures happen only during deploys, the process manager is killing children after its stop timeout (step 5).

signal 11 is a segmentation fault in native code: a bug or incompatibility in a C extension (image processing, XML, PDF libraries). Upgrade or pin the library, and isolate such tasks on their own queue.

2. Measure memory per task

Watch the worker's children while the problematic task runs:

watch -n 2 'ps -o pid,ppid,rss,cmd -u celery --sort=-rss | head -15'

RSS in kilobytes per child shows which tasks balloon. Common patterns in Django tasks:

• Loading a whole queryset into memory: list(Order.objects.all()), or iterating a queryset without .iterator() (the queryset caches every row).
• Building a large file (CSV, PDF, ZIP) entirely in memory.
• Accumulating results in a list during a long loop.
• Libraries that leak memory, so each task leaves the child larger than before.

3. Reduce memory in the task

import csv
import tempfile

from django.core.files import File


@app.task
def export_orders(export_id):
    export = Export.objects.get(pk=export_id)
    qs = Order.objects.filter(created__gte=export.since).values_list("id", "total", "created")
    with tempfile.TemporaryFile(mode="w+", newline="") as fh:
        writer = csv.writer(fh)
        for row in qs.iterator(chunk_size=2000):
            writer.writerow(row)
        fh.seek(0)
        export.file.save(f"orders-{export.pk}.csv", File(fh))

• .iterator(chunk_size=...) streams rows instead of caching the whole result.
• values_list avoids building model instances when you only need columns.
• Writing to a temporary file on disk, then saving it to storage, keeps the output out of memory.

For very large jobs, split the work into many smaller tasks (for example one per batch of IDs) so no single child needs much memory.

4. Cap memory and recycle children

celery -A config worker --concurrency=4 --max-memory-per-child=400000 --max-tasks-per-child=200

• --max-memory-per-child (in kilobytes) replaces a child after it finishes a task if its resident memory exceeds the limit. It does not interrupt the running task; it prevents slow growth from reaching the OOM killer.
• --max-tasks-per-child replaces children after a number of tasks, which also contains leaks.
• Lower --concurrency if the server's memory divided by the worst-case memory per child is below the current concurrency. Four children at 1.5 GB each will not fit on a 4 GB server that also runs Gunicorn.

Also check worker_prefetch_multiplier: prefetched messages do not use much memory themselves, but they matter for the redelivery behaviour in step 6.

5. Stop workers gracefully during deploys

When systemd, Supervisor, Docker, or Kubernetes stops a worker, it sends SIGTERM and then SIGKILL after a timeout. Celery's warm shutdown waits for running tasks; if the timeout is shorter than your tasks, children are killed and their tasks fail with WorkerLostError.

# systemd unit
[Service]
KillSignal=SIGTERM
TimeoutStopSec=600

Equivalent settings are stopwaitsecs in Supervisor, stop_grace_period in Docker Compose, and terminationGracePeriodSeconds in Kubernetes. Pair them with Celery time limits so no task can run longer than the grace period:

CELERY_TASK_SOFT_TIME_LIMIT = 540
CELERY_TASK_TIME_LIMIT = 570

6. Decide what happens to lost tasks

By default, Celery acknowledges a message when a child starts the task, so a lost task is marked failed and not retried. Options:

CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True

With both settings, a task whose child dies is returned to the queue and runs again. That is useful for deploy interruptions, but dangerous for tasks that always exhaust memory: they are redelivered and kill a worker again, forever. Use redelivery only for idempotent tasks, fix memory-hungry tasks first, and monitor for tasks that fail repeatedly.

Verification check:

sudo journalctl -k --since "1 day ago" | grep -ci "killed process"
celery -A config inspect stats | grep -E "max-memory|max-tasks|pool"

Run the previously failing task with production-like data and watch child RSS stay below your per-child limit. No new OOM lines should appear in the kernel log.

Explanation

The prefork pool runs tasks in child processes managed by a parent. The parent detects when a child disappears without reporting a result and raises WorkerLostError for the task it was running. The signal in the message identifies the cause: SIGKILL cannot be caught, so it comes from the kernel (OOM killer, cgroup memory limit) or from a supervisor enforcing a stop timeout. Fixing memory use per task and making stops graceful removes both sources.

Edge cases / notes

Swap and small servers

On servers without swap, memory spikes go straight to the OOM killer. A modest swap file gives headroom for short spikes, but tasks that routinely need more memory than the server has must be restructured.

Separate queues for heavy tasks

Route memory-intensive tasks to a dedicated queue served by a worker with low concurrency and higher memory limits, so they cannot starve lightweight tasks.

When to automate this

Alert on kernel OOM messages and on WorkerLostError in your error tracker, and graph per-child memory. A worker configuration template with sensible --max-memory-per-child, time limits, and stop timeouts prevents most occurrences on new servers.

Next steps

• Set concurrency and limits with Tune Celery Worker Concurrency and Prefetch for Django.
• The same OOM diagnosis applies to web workers in Fix Gunicorn Workers Killed by the OOM Killer.
• Make redelivery safe with Make Celery Tasks Idempotent and Safe to Retry.
• Review worker settings with the Celery Production Checklist for Django.

FAQ

What causes "Worker exited prematurely: signal 9 (SIGKILL)" in Celery?

The child process was killed forcibly, usually by the OOM killer or a memory limit, or by a process manager after its stop timeout.

Does --max-memory-per-child stop a task that uses too much memory?

No. It replaces the child after the current task finishes. A single task that exceeds available memory must be made smaller.

Should I enable task_reject_on_worker_lost?

Only for idempotent tasks, together with acks_late. Otherwise a task that always crashes its worker will be redelivered indefinitely.

Why do WorkerLostError failures happen during deploys?

Workers are stopped with a timeout shorter than the running tasks. Increase the stop timeout and set task time limits below it.