Run Django and Celery with Supervisor

Problem statement

Gunicorn and Celery must be restarted when they crash, started at boot, and stopped cleanly during deploys. systemd does this on most modern Linux servers, but Supervisor is still common: it is used on older servers, inside some containers that run several processes, on shared hosting setups, and by teams who like its single configuration style across distributions.

Most Supervisor problems in Django deployments come from four places: environment variables that the program never receives, stop signals that kill Celery tasks mid-run, logs that grow without limit, and restarts that leave orphaned worker processes. This guide configures Supervisor for Django with those handled.

1. Install Supervisor from the distribution package (apt install supervisor) so it starts at boot.
2. Create one [program:x] section per process in /etc/supervisor/conf.d/myapp.conf.
3. Run each program as the app user with user=, a directory=, and full paths to the virtual environment's binaries.
4. Load secrets through a wrapper script that sources the environment file, rather than hardcoding them in environment=.
5. Set stopsignal=TERM, stopwaitsecs longer than your longest task, and stopasgroup=true/killasgroup=true.
6. Configure log size limits (stdout_logfile_maxbytes, stdout_logfile_backups).
7. Apply changes with supervisorctl reread and supervisorctl update; restart the group on deploys.

Step-by-step solution

1. Install

sudo apt install supervisor
sudo systemctl enable --now supervisor
sudo supervisorctl status

The package includes /etc/supervisor/supervisord.conf, which includes every file in /etc/supervisor/conf.d/.

2. Create a wrapper that loads the environment

Supervisor's environment= line accepts KEY="value" pairs, but putting secrets there means they live in a config file readable by anyone who can read /etc/supervisor/conf.d/, and values containing commas or quotes need escaping. A small wrapper script is cleaner:

#!/usr/bin/env bash
# /srv/myapp/bin/run-with-env
set -euo pipefail
set -a
. /etc/myapp.env
set +a
exec "$@"

sudo chmod 755 /srv/myapp/bin/run-with-env
sudo chown root:myapp /etc/myapp.env && sudo chmod 640 /etc/myapp.env

exec replaces the shell with the real program, so Supervisor's signals go directly to Gunicorn or Celery, not to a shell that ignores them.

3. Define the programs

; /etc/supervisor/conf.d/myapp.conf
[program:myapp-web]
command=/srv/myapp/bin/run-with-env /srv/myapp/venv/bin/gunicorn config.wsgi:application
    --bind unix:/run/myapp/gunicorn.sock --workers 3 --timeout 30 --access-logfile -
directory=/srv/myapp/app
user=myapp
autostart=true
autorestart=true
startsecs=5
stopsignal=TERM
stopwaitsecs=30
stopasgroup=true
killasgroup=true
redirect_stderr=true
stdout_logfile=/var/log/myapp/web.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=5

[program:myapp-worker]
command=/srv/myapp/bin/run-with-env /srv/myapp/venv/bin/celery -A config worker
    --loglevel=INFO --concurrency=4
directory=/srv/myapp/app
user=myapp
autostart=true
autorestart=true
startsecs=10
stopsignal=TERM
stopwaitsecs=600
stopasgroup=true
killasgroup=true
redirect_stderr=true
stdout_logfile=/var/log/myapp/worker.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=5

[program:myapp-beat]
command=/srv/myapp/bin/run-with-env /srv/myapp/venv/bin/celery -A config beat
    --loglevel=INFO --schedule=/var/lib/myapp/celerybeat-schedule
directory=/srv/myapp/app
user=myapp
autostart=true
autorestart=true
numprocs=1
stopsignal=TERM
stopwaitsecs=30
redirect_stderr=true
stdout_logfile=/var/log/myapp/beat.log
stdout_logfile_maxbytes=20MB
stdout_logfile_backups=3

[group:myapp]
programs=myapp-web,myapp-worker,myapp-beat

Key settings:

• Continuation lines: Supervisor allows a value to continue on indented lines, which keeps long commands readable.
• stopsignal=TERM and stopwaitsecs: Celery's warm shutdown on SIGTERM stops taking new tasks and waits for running ones. stopwaitsecs is how long Supervisor waits before SIGKILL; if it is shorter than your tasks, they are killed mid-run.
• stopasgroup and killasgroup: send signals to the whole process group, so Gunicorn and Celery child processes are not left orphaned.
• startsecs: the program must stay up this long to count as started; a crash loop is then reported as FATAL instead of restarting forever.
• Beat: exactly one instance. Running Beat on two servers sends every scheduled task twice.

Create the directories the programs need:

sudo mkdir -p /var/log/myapp /var/lib/myapp /run/myapp
sudo chown myapp:myapp /var/lib/myapp /run/myapp
sudo chown myapp:adm /var/log/myapp

/run is a temporary filesystem, so /run/myapp disappears at reboot. Recreate it at boot with a systemd-tmpfiles rule:

# /etc/tmpfiles.d/myapp.conf
d /run/myapp 0755 myapp www-data -

4. Load the configuration

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl status

reread parses changed files; update applies them, starting new programs and restarting changed ones. reload restarts supervisord itself and every program, which is rarely what you want.

5. Deploy with Supervisor

cd /srv/myapp/app && git pull
/srv/myapp/venv/bin/pip install -r requirements.txt
/srv/myapp/bin/run-with-env /srv/myapp/venv/bin/python manage.py migrate --noinput
/srv/myapp/bin/run-with-env /srv/myapp/venv/bin/python manage.py collectstatic --noinput
sudo supervisorctl restart myapp:*

For Gunicorn only, a graceful reload keeps accepting connections while workers are replaced:

sudo supervisorctl signal HUP myapp:myapp-web

Celery workers need a full restart to load new code.

Verification check:

sudo supervisorctl status myapp:*
tail -n 50 /var/log/myapp/web.log
curl -sI https://example.com/ | head -1

All programs should show RUNNING with an uptime that matches the deploy time.

Explanation

supervisord runs as root, starts each program as the configured user, and watches the processes it started. When one exits unexpectedly, it restarts it according to autorestart; when you stop it, it sends stopsignal, waits stopwaitsecs, then kills it. Because Supervisor only knows about the processes it started directly, the exec in the wrapper and the group signal settings are what make it manage Gunicorn and Celery correctly rather than a shell or half of a process tree.

Edge cases / notes

Supervisor inside containers

Running Supervisor as PID 1 to start several processes in one container works, but it hides failures from the container runtime (the container stays up while a program is FATAL). Prefer one process per container and let the orchestrator restart it.

systemd instead

On systemd-based servers, native units give dependency ordering, resource limits (MemoryMax), journald logging, and security sandboxing options that Supervisor lacks. New deployments on such servers usually use systemd; existing Supervisor setups work fine and do not need to be migrated urgently.

When to automate this

Template the program file and wrapper in your provisioning tool, and make the deploy script call supervisorctl restart myapp:* and check status afterwards. Alert when any program is not RUNNING, for example with a scheduled supervisorctl status check.

Next steps

• Compare with native units in How to Configure systemd for Gunicorn and Django.
• See worker-specific supervision in Run Celery Workers with systemd for Django.
• Keep secrets out of config files with Manage Django Secrets in Production.
• If Gunicorn keeps exiting, work through Fix Gunicorn "Worker failed to boot".

FAQ

Why doesn't my Django app see environment variables under Supervisor?

Supervisor does not read shell profiles or .env files. Pass variables with environment= or source an environment file in a wrapper script that execs the program.

Why are Celery tasks killed when Supervisor restarts the worker?

stopwaitsecs is shorter than the tasks, so Supervisor sends SIGKILL before the warm shutdown finishes. Increase it and make tasks safe to retry.

What is the difference between supervisorctl reread, update, and reload?

reread detects changed config files, update applies those changes to affected programs, and reload restarts supervisord and every program.

Should I use Supervisor or systemd for Django?

On modern Linux servers, systemd is the default and offers more features. Supervisor is fine where it is already in use or where systemd is not available.