Fix "Too many open files" on a Django Server

Problem statement

Under load, or after the service has been running for a while, requests start failing with:

OSError: [Errno 24] Too many open files

or in the Nginx error log:

accept4() failed (24: Too many open files)
socket() failed (24: Too many open files) while connecting to upstream

Every open file, socket, pipe, and connection uses a file descriptor, and each process has a limit. When a Django worker, Celery worker, or Nginx reaches it, new connections and file operations fail. There are two possibilities: the limit is too low for legitimate load (common, because service defaults are low), or something leaks descriptors and would exhaust any limit eventually. This guide measures which one and fixes it.

1. Read the running process's limit: grep "open files" /proc/<pid>/limits.
2. Count its open descriptors now: ls /proc/<pid>/fd | wc -l, and repeat over time.
3. If usage is stable but close to the limit, raise it: LimitNOFILE=65536 in the systemd unit, worker_rlimit_nofile in Nginx, ulimits in Docker.
4. If usage grows steadily, find the leak with lsof -p <pid> grouped by type and target.
5. Fix leaks in code: close files, reuse HTTP sessions and clients, and avoid creating connections per call.
6. Restart the service and monitor the descriptor count.

Step-by-step solution

1. Check the real limit

ulimit -n in your shell shows your shell's limit, not the service's. Look at the process:

PID=$(systemctl show -p MainPID --value gunicorn)
grep -E "Max open files" /proc/$PID/limits
ls /proc/$PID/fd | wc -l

For Gunicorn and Celery, check the worker processes too (children of the master):

for p in $(pgrep -P $PID); do echo "$p $(ls /proc/$p/fd | wc -l)"; done

Services started by systemd often run with a soft limit of 1024 descriptors unless the unit sets LimitNOFILE. That is enough for a quiet site and too low for busy Nginx or async workers.

2. Decide: load or leak

Sample the count every minute for a while:

while true; do echo "$(date +%T) $(ls /proc/$PID/fd | wc -l)"; sleep 60; done

• Stable, rises with traffic, falls when traffic drops: legitimate use; raise the limit (step 3).
• Grows steadily regardless of traffic and resets only on restart: a leak (step 4).

3. Raise limits where needed

systemd services (Gunicorn, Celery, Daphne, Uvicorn):

# /etc/systemd/system/gunicorn.service
[Service]
LimitNOFILE=65536

sudo systemctl daemon-reload && sudo systemctl restart gunicorn
grep "open files" /proc/$(systemctl show -p MainPID --value gunicorn)/limits

Nginx. Each proxied request uses two descriptors (client and upstream), plus files it serves:

# /etc/nginx/nginx.conf (main context)
worker_rlimit_nofile 65536;

events {
    worker_connections 16384;
}

worker_connections counts all connections per worker, including upstream ones; worker_rlimit_nofile must be comfortably above it. If Nginx runs under systemd, LimitNOFILE in its unit (via a drop-in) also applies.

Docker:

services:
  web:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

Kernel-wide limit: cat /proc/sys/fs/file-max is usually very large on modern systems and rarely the constraint.

/etc/security/limits.conf applies to login sessions through PAM, not to systemd services, which is why editing it often appears to have no effect.

4. Find a leak

sudo lsof -p $PID -n -P | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn
sudo lsof -p $PID -n -P | awk 'NR>1 && $5=="REG" {print $9}' | sort | uniq -c | sort -rn | head
sudo lsof -p $PID -n -P -i | awk 'NR>1 {print $9}' | sed 's/->.*:/-> :/' | sort | uniq -c | sort -rn | head

The first command groups descriptors by type (REG files, IPv4/IPv6 sockets, unix sockets, FIFO pipes). The second shows which regular files are open most often; the third which remote endpoints sockets point at.

Common findings in Django applications:

• Thousands of the same file (a template, a certificate bundle, a log file): code opening a file without closing it, often open(path).read() in a loop, or a logging handler created per request.
• Many sockets to an external API: a new HTTP client or requests.Session created per call and never closed, or a library pool that grows without bound. Create one session per process and reuse it.
• Many sockets to PostgreSQL: connections opened in threads or background loops that never close them. Django closes its own per-request connections; custom threads must call django.db.connection.close() when done.
• Pipes (FIFO): subprocesses started without waiting for them or closing their pipes.

5. Fix leaks in code

# Leaks one descriptor per call until garbage collection gets around to it
data = json.load(open(path))

# Closes deterministically
with open(path, encoding="utf-8") as fh:
    data = json.load(fh)

# One session per process, reused across requests
import requests

_session = None


def http():
    global _session
    if _session is None:
        _session = requests.Session()
    return _session

import subprocess

result = subprocess.run(["convert", src, dst], check=True, capture_output=True, timeout=60)

subprocess.run waits for the process and closes its pipes; Popen without communicate() or wait() leaks them.

As a stopgap while fixing a leak, Gunicorn's --max-requests with jitter recycles workers periodically, releasing their descriptors. It contains the leak; it does not fix it.

Verification check:

grep "open files" /proc/$(systemctl show -p MainPID --value gunicorn)/limits
ls /proc/$(systemctl show -p MainPID --value gunicorn)/fd | wc -l
sudo grep -c "Too many open files" /var/log/nginx/error.log

The limit shows the new value, the descriptor count stays well below it under peak load, and no new errors appear.

Explanation

On Linux, sockets, files, and pipes are all file descriptors, and RLIMIT_NOFILE caps how many a process may hold. Services inherit their limits from systemd (or the container runtime), not from shell settings. A proxy doubles descriptor use per request, async servers hold many connections per process, and each leak accumulates until the cap is hit. Measuring usage over time separates a low limit (raise it) from a leak (fix it), which prevents the common mistake of raising the limit and only delaying the next outage.

Edge cases / notes

inotify limits

OSError: [Errno 28] inotify watch limit reached or Too many open files from file watchers comes from fs.inotify.max_user_watches or max_user_instances, not RLIMIT_NOFILE. It affects development reloaders and some tools, rarely production Django.

PostgreSQL and Redis

Database servers have their own descriptor needs. PostgreSQL uses max_files_per_process; Redis lowers maxclients automatically if its descriptor limit is too low and logs a warning at start-up.

When to automate this

Put LimitNOFILE in your unit templates and worker_rlimit_nofile in the Nginx template, and export the open descriptor count per process as a metric with an alert at 80 percent of the limit. Leaks then show up as a rising line days before they cause errors.

Next steps

• Add the limit to your units with How to Configure systemd for Gunicorn and Django.
• Size Nginx workers with How to Configure Nginx for Django in Production.
• If the database runs out of connections instead, see Fix PostgreSQL "Too Many Connections" in Django.
• Plan resources with Size a Server for a Django App: Capacity Planning.

FAQ

Why does ulimit -n show a high value but Gunicorn still fails?

ulimit shows your shell's limit. systemd services get their limits from the unit's LimitNOFILE. Check /proc/<pid>/limits.

What LimitNOFILE should I set for Django services?

65536 is a common, safe value for web and worker services. The right number is comfortably above the peak observed descriptor count.

Why does Nginx run out of file descriptors?

Each proxied request uses a client and an upstream descriptor. Raise worker_rlimit_nofile above worker_connections, and the systemd limit if Nginx runs under systemd.

How do I know if my Django app leaks file descriptors?

The descriptor count grows steadily over time regardless of traffic and resets on restart. lsof -p <pid> shows which files or sockets accumulate.