Reduce Django Memory Usage per Worker

Problem statement

Each Gunicorn worker and Celery child process uses 200 to 400 MB, so a server that should run eight workers fits only four, or containers get OOM-killed under load. Memory in Django processes comes from three places: what is imported at startup, what requests allocate temporarily, and what never gets released because CPython keeps freed memory for reuse. Each has different fixes.

1. Measure RSS per worker after warm-up and at peak (ps -o pid,rss,cmd -C gunicorn).
2. Reduce request peaks: .iterator() for big querysets, .only()/.values() for fewer columns, streaming responses for exports.
3. Recycle workers with --max-requests and jitter (Gunicorn) or --max-tasks-per-child (Celery).
4. Trim the baseline: remove unused apps and heavy imports at module level; import heavy libraries lazily.
5. Consider jemalloc (LD_PRELOAD) to reduce fragmentation, then size worker counts from measured numbers.

Step-by-step solution

1. Measure the right numbers

ps -o pid,rss,etime,cmd -C gunicorn --sort=-rss | head

Note the baseline RSS of a freshly started worker, after a few hundred requests (warm), and after hours (high-water). A worker that grows steadily without leveling off has a leak; see Fix Django Memory Leak in Production. A worker that jumps on specific endpoints has peak allocations to fix.

2. Cut request peaks

Loading a large queryset materializes every row as a model instance:

# 500k Order instances in memory
for order in Order.objects.filter(year=2025):
    write_row(order)

# constant memory, chunks of 2000 rows
for order in Order.objects.filter(year=2025).only("id", "total", "created_at").iterator(chunk_size=2000):
    write_row(order)

Other common peaks: building a full CSV or JSON response in memory (stream it), reading uploaded files fully into memory (let Django spool large uploads to disk, keep FILE_UPLOAD_MAX_MEMORY_SIZE modest), and Pillow operations on huge images (limit dimensions, process in Celery).

3. Recycle workers

CPython rarely returns freed memory to the operating system, so a worker's RSS stays at its highest point. Restarting workers periodically resets it:

# gunicorn.conf.py
max_requests = 1000
max_requests_jitter = 200

celery -A config worker --concurrency 4 --max-tasks-per-child 200 --max-memory-per-child 400000

--max-memory-per-child (in KiB) replaces a Celery child after its current task once it exceeds the limit.

4. Trim the baseline

python -X importtime manage.py check 2>&1 | sort -t'|' -k2 -n | tail -20

This lists the slowest (often largest) imports. Typical savings: remove unused apps from INSTALLED_APPS, move imports of pandas, NumPy, or ML libraries into the functions that need them, and avoid loading large data files at module level. Gunicorn's --preload can share read-only memory between workers through copy-on-write, though reference counting limits how much stays shared.

5. Try jemalloc and size workers

Some workloads fragment memory with glibc's allocator. jemalloc often reduces RSS:

RUN apt-get update && apt-get install -y --no-install-recommends libjemalloc2 && rm -rf /var/lib/apt/lists/*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

Measure before and after; results vary. Then size the deployment: available memory, minus headroom for the OS and other processes, divided by warm per-worker RSS, gives the worker count. If that is too low for your concurrency, use gthread workers with several threads each.

Explanation

A Django worker's memory is the interpreter plus imported modules (baseline), plus objects created while handling requests. CPython's allocator (pymalloc) and the C library allocator keep freed memory in pools for reuse, so peak allocations raise RSS permanently even when objects are freed. That is why fixing peaks and recycling workers matter more than chasing every object. Process-based servers multiply per-worker memory by the worker count, which makes per-worker savings directly increase capacity.

Edge cases / notes

• DEBUG = True. Django stores every SQL query in memory per request with debug on; never run production with it.
• Caches in globals. Module-level dictionaries used as caches grow without bound; use the cache framework or lru_cache(maxsize=...).
• Containers. Python sees the host's memory, not the container limit; set worker counts explicitly rather than from CPU or memory autodetection.

Next steps

• Distinguish leaks from peaks with Fix Django Memory Leak in Production.
• Size workers with Optimize Gunicorn Workers for Django.
• Share memory carefully with Gunicorn Preload App for Django.

FAQ

Why does memory not go down after a big request?

CPython keeps freed memory for reuse instead of returning it to the OS. Recycle workers to reset RSS.

Does max_requests hide memory leaks?

It contains them. If memory grows quickly between restarts, find and fix the leak.

Will jemalloc always help?

Not always. It often reduces fragmentation; measure with your workload.

Threads or processes to save memory?

Threads share one process's memory, so gthread workers can handle more concurrent I/O-bound requests per MB.