Fix a Full Disk on a Django Production Server

Problem statement

A Django site starts failing in unrelated-looking ways:

OSError: [Errno 28] No space left on device
django.db.utils.OperationalError: could not extend file "base/16384/2619": No space left on device
PANIC:  could not write to file "pg_wal/xlogtemp.1234": No space left on device

Nginx logs may stop, Celery tasks fail, uploads return 500, collectstatic and pip install fail during deploys, and PostgreSQL can shut down to protect its data. The root cause is a filesystem with no free space (or no free inodes).

The pressure is to delete things quickly. The risk is deleting something that matters: database files, backups, or files a running process still holds open, which frees no space at all. This guide recovers space in a safe order and then prevents recurrence.

Quick answer

1. Confirm which filesystem is full: df -h and df -i (inodes).
2. Find the largest directories on it: sudo du -xh --max-depth=1 / | sort -h.
3. Free space in safe categories first: systemd journal (journalctl --vacuum-size), rotated logs, package caches, Docker build cache and unused images, old release directories.
4. Check for deleted files still held open: sudo lsof +L1; restart or truncate the owning process's log.
5. Never delete files inside the PostgreSQL data directory, including pg_wal, by hand.
6. Restart failed services (PostgreSQL, Gunicorn, Celery) and verify.
7. Fix the cause: log rotation, retention, Docker pruning, backup placement, and disk usage alerts.

Step-by-step solution

1. Identify the full filesystem

df -h
df -i

• Use% 100% in df -h means no free blocks.
• IUse% 100% in df -i means no free inodes, even if bytes are available. Millions of tiny files (session files, cache files, mail spool) cause this.

Note the mount point. /, /var, and a separate data volume for PostgreSQL or Docker behave differently.

2. Find where the space went

Stay on the same filesystem with -x so you do not descend into other mounts:

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -15
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -15

Drill down into the largest directory until you find the specific cause. ncdu makes this interactive if installed. Common culprits on Django servers:

For inode exhaustion, count files per directory:

sudo find /var -xdev -type f | cut -d/ -f2-4 | sort | uniq -c | sort -n | tail

3. Free space safely: journal and logs

The systemd journal can be vacuumed without affecting services:

journalctl --disk-usage
sudo journalctl --vacuum-size=500M

Rotated and compressed logs can be removed:

sudo find /var/log -type f \( -name "*.gz" -o -name "*.[0-9]" \) -mtime +7 -print -delete

For a huge active log file, truncate instead of deleting it, so the writing process keeps a valid file:

sudo truncate -s 0 /var/log/myapp/app.log

Deleting an open log file with rm removes its name, but the process keeps writing to the now-invisible file and the space is not freed until the process closes it (step 5).

4. Free space safely: Docker

docker system df
docker builder prune --force
docker image prune --all --filter "until=168h" --force
docker container prune --force

• Build cache and dangling images are safe to remove.
• image prune --all removes images not used by any container, including images you might roll back to; the until filter keeps recent ones.
• Do not run docker volume prune unless you are sure: volumes hold database data and uploads.

Container logs with the default json-file driver grow without limit. Check their size:

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail

Configure rotation in /etc/docker/daemon.json (applies to newly created containers):

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "3" }
}

5. Deleted files still held open

sudo lsof +L1 2>/dev/null | awk 'NR==1 || $7 > 100000000'

This lists open files with no remaining directory entry, showing the process and size. Free the space by restarting the process (systemctl restart gunicorn), or for a log file you cannot restart immediately, truncate it through the process's file descriptor:

sudo truncate -s 0 /proc/<PID>/fd/<FD>

6. PostgreSQL: do not delete WAL files

If /var/lib/postgresql is large, find out why before touching anything:

SELECT pg_size_pretty(pg_database_size(current_database()));
SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots;

• An inactive replication slot retains WAL indefinitely. Drop it only if the replica or tool using it is gone: SELECT pg_drop_replication_slot('slot_name');
• A failing archive_command makes PostgreSQL keep WAL until archiving succeeds; fix the archive destination.
• Large tables may need cleanup of old data or VACUUM FULL (which needs extra free space temporarily and locks the table).

Deleting files from pg_wal or base by hand can corrupt the database. If PostgreSQL has already stopped because of the full disk, free space elsewhere on the filesystem first, then start it.

7. Old releases and backups

Keep a bounded number of release directories:

cd /srv/myapp/releases
ls -1t | tail -n +6
ls -1t | tail -n +6 | xargs -r rm -rf

Review the list before running the delete, and make sure the current symlink does not point at a directory being removed. Move backups off the server: a backup on the same disk does not protect against disk failure and consumes the space production needs.

8. Restart and verify services

df -h
sudo systemctl status postgresql nginx gunicorn celery-worker --no-pager | grep -E "●|Active"
sudo systemctl restart gunicorn celery-worker
curl -sI https://www.example.com/ | head -1

Check PostgreSQL logs after recovery for errors, and run a quick application smoke test including an upload.

Verification check:

• the filesystem has comfortable free space (for example more than 20 percent)
• PostgreSQL, Gunicorn, Celery, and Nginx are active
• the site responds and uploads succeed
• lsof +L1 shows no large deleted-but-open files

9. Prevent recurrence

• Journal limits: set SystemMaxUse=1G in /etc/systemd/journald.conf and restart systemd-journald.
• Log rotation: logrotate for application logs, with compress and a retention count.
• Docker log limits: the daemon.json settings above, and scheduled docker builder prune.
• Release retention: prune old releases as part of each deploy.
• Backup placement: ship backups to object storage; keep at most one local copy.
• Monitoring: alert when any filesystem exceeds 80 percent usage or inode usage, and alert on disk growth rate so you notice trends before they become incidents.

Explanation

A full disk breaks every component that writes: databases, logs, uploads, temporary files, and package installs. The recovery order matters because some files are cheap and safe to remove (journal entries, rotated logs, build caches) while others are critical (database files, volumes) or ineffective to delete (files held open by running processes). Measuring first, cleaning safe categories, and handling open files correctly restores service without creating a second incident.

Edge cases / notes

Reserved blocks on ext4

ext4 reserves a percentage of blocks for root by default. Services running as non-root users may fail while df still shows a little space. Do not rely on the reserve; it is there to let administrators recover.

Media growth is legitimate

If uploads are the largest consumer, the disk is simply too small for the product's data. Move media to object storage or attach a larger volume, rather than deleting user files.

Temporary upload files

Large uploads are streamed to FILE_UPLOAD_TEMP_DIR or the system temp directory. Interrupted requests can leave files behind; PrivateTmp=true in the service unit isolates and cleans them when the service restarts.

Common mistakes

• Running rm on a large active log and seeing no space freed.
• Running docker system prune --volumes on a server with database volumes.
• Deleting files in pg_wal to recover space.
• Keeping backups only on the same disk as the database.

Next steps

• Configure rotation properly with How to Rotate Django and Gunicorn Logs on Linux.
• Add disk alerts with Set Up Uptime Monitoring and Alerts for Django.
• Reduce image and build cache size with Build Smaller, Faster Django Docker Images.
• Keep backups safe and off-server with Backup and Restore PostgreSQL for Django Apps.

FAQ

How do I find what is filling the disk on a Linux server?

Run df -h to find the full filesystem, then sudo du -xh --max-depth=1 <mount> | sort -h repeatedly on the largest directories until you find the cause.

Why didn't deleting a log file free disk space?

A running process still has the file open. The space is released only when the process closes it. Restart the process or truncate the file through /proc/<PID>/fd/<FD>.

Is it safe to run docker system prune on a Django server?

Without --volumes, it removes stopped containers, unused networks, dangling images, and build cache. With --volumes, it can delete database and media data, so avoid that flag unless you are certain.

Can I delete PostgreSQL WAL files to free space?

No. Manually deleting WAL files can corrupt the database. Find out why WAL is retained, such as an inactive replication slot or failing archiving, and fix that cause.