Fix Gunicorn Workers Killed by the OOM Killer

Problem statement

Requests fail intermittently with 502 Bad Gateway, and Gunicorn's log shows workers disappearing:

[ERROR] Worker (pid:23114) was sent SIGKILL! Perhaps out of memory?
[INFO] Booting worker with pid: 23188

In containers, the symptom is a restart with exit code 137 or a Kubernetes status of OOMKilled. On a VM, the kernel log contains lines such as:

Out of memory: Killed process 23114 (gunicorn) total-vm:1432100kB, anon-rss:812340kB

A process killed with SIGKILL cannot clean up or log anything itself, so Django and Gunicorn show no Python traceback. The request that was in progress simply fails. This is different from a Gunicorn WORKER TIMEOUT, where Gunicorn itself kills a silent worker and logs the timeout.

The kernel (or a container runtime's memory limit) kills a process when the machine or cgroup runs out of memory. The Gunicorn worker is often chosen because it is the largest process, but it may not be the only cause.

Quick answer

1. Confirm an OOM kill: check dmesg or journalctl -k, systemctl status for oom-kill, or the container's OOMKilled state.
2. Determine whether it is a system-wide OOM (host out of memory) or a cgroup limit (MemoryMax, Docker --memory, Kubernetes limits).
3. Measure resident memory per worker at idle and under load.
4. Reduce memory demand: fewer workers per host, --max-requests with jitter to recycle workers, and threads instead of extra processes for I/O-bound apps.
5. Find memory spikes from specific endpoints: large querysets, file processing, report generation; fix with iteration, streaming, or background jobs.
6. Investigate true leaks with tracemalloc or memray in staging.
7. Set limits and alerts so one service cannot take down the host.

Step-by-step solution

1. Confirm the OOM kill

VM with systemd:

sudo journalctl -k --since "2 hours ago" | grep -i -E "out of memory|oom-kill|killed process"
systemctl status gunicorn --no-pager | grep -i oom

Lines mentioning oom-kill:constraint=CONSTRAINT_MEMCG indicate a cgroup limit (for example MemoryMax= on the unit). CONSTRAINT_NONE indicates the whole host ran out of memory.

Docker:

docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' <container>
docker stats --no-stream

Kubernetes:

kubectl describe pod <pod> | grep -A3 "Last State"

If the whole container is killed (not just a worker), the Gunicorn master died too, so the restart comes from Docker or Kubernetes rather than Gunicorn.

2. Measure memory per worker

ps -o pid,ppid,rss,vsz,etime,cmd -C gunicorn --sort=-rss

RSS (resident set size, in kilobytes) is the physical memory used. Sum it across workers for a rough total; shared pages are counted in each process, so the sum overstates real usage somewhat. For a more accurate view, smem reports proportional set size (PSS), which divides shared pages fairly:

sudo smem -k -P gunicorn

Record three numbers:

• memory per worker right after boot
• memory per worker after an hour of normal traffic
• peak memory during the heaviest endpoints

A worker that grows steadily over hours suggests a leak or unbounded cache. A worker that jumps on specific requests suggests an expensive endpoint.

3. Compare with available memory

free -m
cat /sys/fs/cgroup/system.slice/gunicorn.service/memory.max 2>/dev/null

Do the math:

(workers × peak MB per worker) + master + Nginx + PostgreSQL + Redis + Celery + OS cache headroom

If that exceeds available memory, OOM kills are expected under load. The Gunicorn documentation's starting formula of 2 × CPU cores + 1 workers ignores memory entirely, so on small, memory-limited servers it often produces too many workers.

4. Right-size workers and threads

For I/O-bound Django apps (most apps that wait on the database and HTTP APIs), threads add concurrency with little extra memory:

gunicorn config.wsgi:application \
  --workers 2 \
  --threads 4 \
  --worker-class gthread \
  --max-requests 1000 \
  --max-requests-jitter 100

• Fewer processes with more threads uses less memory for the same concurrency.
• --max-requests restarts each worker after that many requests, which caps slow memory growth from fragmentation or leaks. The jitter staggers restarts so workers do not all recycle at once.

Recycling workers is a mitigation. If memory grows quickly, find the cause.

5. Find memory-heavy endpoints

Common Django patterns that load too much into memory:

Iterating large querysets:

# Loads every row into the queryset cache
for order in Order.objects.all():
    export_row(order)

# Streams rows in chunks without caching them
for order in Order.objects.all().iterator(chunk_size=2000):
    export_row(order)

Building large responses in memory:

import csv

from django.http import StreamingHttpResponse


class Echo:
    def write(self, value):
        return value


def export_orders(request):
    writer = csv.writer(Echo())
    rows = (
        writer.writerow([o.pk, o.created_at, o.total])
        for o in Order.objects.values_list("pk", "created_at", "total").iterator()
    )
    return StreamingHttpResponse(rows, content_type="text/csv")

Reading uploads fully: request.FILES["file"].read() on a large upload loads it all. Process in chunks with for chunk in upload.chunks():.

Image and document processing: resizing large images or generating PDFs can briefly use hundreds of megabytes. Move this work to Celery workers with their own memory limits.

To find which requests coincide with kills, correlate OOM timestamps with access logs for requests in progress at that moment, looking for long $request_time values.

6. Investigate a leak

Suspect a leak when memory grows steadily with ordinary traffic. Common causes:

• module-level dictionaries or lists used as caches without size limits
• DEBUG = True in production, which stores every SQL query in connection.queries
• libraries that accumulate state per request (misconfigured instrumentation, logging handlers added repeatedly)

In staging, use Python's built-in tracemalloc to compare snapshots:

import tracemalloc

tracemalloc.start(25)
snapshot_before = tracemalloc.take_snapshot()
# ... run a batch of representative requests with the test client ...
snapshot_after = tracemalloc.take_snapshot()

for stat in snapshot_after.compare_to(snapshot_before, "lineno")[:15]:
    print(stat)

The largest positive differences point at the lines allocating memory that was not released. Tools such as memray give more detailed flame graphs of allocations when you need them.

7. Set limits deliberately

Make memory failures contained and visible:

# systemd unit
[Service]
MemoryHigh=1400M
MemoryMax=1600M

MemoryHigh throttles and reclaims memory before the hard MemoryMax limit triggers the OOM killer. Protect critical services such as PostgreSQL from being chosen by the host-level OOM killer:

# PostgreSQL drop-in (systemctl edit postgresql)
[Service]
OOMScoreAdjust=-900

Add monitoring and alerting on memory usage per service and on OOM events, so kills are noticed before users report 502 errors.

8. Swap considerations

A small amount of swap can absorb short spikes on small VMs and prevent immediate OOM kills, but heavy swapping makes Django extremely slow. If you add swap, alert on sustained swap usage and treat it as a capacity signal rather than a fix.

9. Verify

sudo journalctl -k --since "24 hours ago" | grep -c -i "killed process"
ps -o pid,rss,etime,cmd -C gunicorn --sort=-rss | head

Verification check:

• no new OOM kills over a full traffic cycle (including peak hours and scheduled jobs)
• worker memory stays within the budget you calculated
• memory-heavy endpoints stream or run in the background
• alerts fire in staging when you deliberately exceed a test memory limit

Explanation

Each Gunicorn worker process holds its own copy of the Django application, imported modules, and any per-process caches. Total memory therefore scales linearly with the worker count, while request patterns add spikes on top. The Linux OOM killer protects the system by killing a large process when allocations cannot be satisfied, and in containers or cgroups it enforces the configured limit. Fixing OOM kills means making memory demand predictable: fewer, well-sized workers, bounded per-request memory, and background processing for heavy work.

Edge cases / notes

--preload and copy-on-write

--preload imports the app once in the master before forking. Some memory is shared between workers until modified, but Python's reference counting touches many objects, so savings are usually modest. It also changes reload behavior, so measure before relying on it.

OOM kills of Celery workers

Celery worker processes can hit the same issue. Use --max-memory-per-child (in kilobytes) to replace child processes that exceed a memory threshold after they finish their current task.

Memory limits on platforms

PaaS instances have fixed memory. If a platform restarts your app for memory, reduce WEB_CONCURRENCY or the equivalent worker setting before upgrading to a larger instance.

Common mistakes

• Using 2 × cores + 1 workers on a 1 GB server.
• Increasing the timeout when the real problem is SIGKILL from memory exhaustion.
• Loading entire tables into memory for exports.
• Relying only on --max-requests while a leak grows within a few hundred requests.

Next steps

• Tune worker counts with How to Optimize Gunicorn Workers for Django.
• Distinguish timeouts from memory kills in Gunicorn Worker Timeout in Django: Root Causes and Fixes.
• Plan resources with Size a Server for a Django App: Capacity Planning.
• Find hot code with Profile Slow Django Endpoints in Production with py-spy.

FAQ

What does "Worker was sent SIGKILL! Perhaps out of memory?" mean in Gunicorn?

A worker was killed with a signal that cannot be caught, most commonly by the Linux OOM killer or a container memory limit. Check the kernel log and container status to confirm.

How much memory does each Gunicorn worker use for Django?

It depends on your code and dependencies. Many Django apps use roughly 60 to 200 MB per worker at idle, and more under load. Measure RSS or PSS on your own deployment.

Does --max-requests fix Gunicorn memory leaks?

It limits the impact by periodically replacing workers, which caps slow growth. It does not remove the cause, and fast-growing leaks can still trigger OOM kills between restarts.

Should I use threads or more workers to reduce memory?

For I/O-bound Django apps, fewer workers with several threads each provide concurrency with less memory. CPU-heavy work benefits more from processes, which cost more memory.