Graceful Shutdown for Django and Celery (SIGTERM)

Problem statement

Every deploy, scale-down, and node drain stops processes. If they stop abruptly, in-flight requests fail with 502s, uploads break midway, and Celery tasks are interrupted and either lost or run again. Graceful shutdown means: stop accepting new work, finish what is in progress within a time limit, then exit. Gunicorn, Uvicorn, and Celery all support it on SIGTERM, but only if the signal actually reaches them, the platform waits long enough, and the load balancer stops sending traffic first.

1. Make the server the process that receives SIGTERM: exec in entrypoint scripts, exec-form CMD, or an init like tini.
2. Gunicorn: graceful_timeout (default 30 s) is how long workers get to finish after SIGTERM.
3. Celery: SIGTERM triggers warm shutdown (finish running tasks); use acks_late with idempotent tasks for anything interrupted.
4. Docker: stop_grace_period longer than the app's graceful timeout (default is 10 s).
5. Kubernetes: terminationGracePeriodSeconds above the graceful timeout plus a short preStop sleep so endpoints update before shutdown.
6. systemd: KillSignal=SIGTERM and TimeoutStopSec above the graceful timeout.

Step-by-step solution

1. Make sure the signal arrives

In containers, the main process is PID 1. A shell running Gunicorn as a child does not forward SIGTERM:

CMD gunicorn config.wsgi:application          # shell form: sh is PID 1, signal lost
CMD ["gunicorn", "config.wsgi:application"]   # exec form: gunicorn is PID 1

Entrypoint scripts must end with exec "$@". If the app spawns subprocesses, run with docker run --init or tini as PID 1 to forward signals and reap zombies.

2. Gunicorn

On SIGTERM, the Gunicorn master stops accepting connections and tells workers to finish current requests, waiting up to graceful_timeout, then kills stragglers.

# gunicorn.conf.py
graceful_timeout = 30
timeout = 60

Set graceful_timeout near your longest normal request. Uvicorn workers behave similarly (--timeout-graceful-shutdown).

3. Kubernetes

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: web
      lifecycle:
        preStop:
          exec: { command: ["sleep", "10"] }

When a pod terminates, removal from Service endpoints and the preStop/SIGTERM sequence happen in parallel. Load balancers and kube-proxy take a few seconds to stop routing to the pod. The preStop sleep keeps the pod serving during that window, so no new requests hit a server that has stopped accepting. Newer Kubernetes versions also offer a native sleep action for preStop.

4. Celery workers

On SIGTERM, a Celery worker stops consuming new messages and waits for running tasks to complete (warm shutdown). A second SIGTERM or SIGQUIT forces a cold shutdown. Give workers a grace period longer than typical task duration:

terminationGracePeriodSeconds: 600

# systemd
KillSignal=SIGTERM
TimeoutStopSec=600

For tasks longer than any reasonable grace period, use acks_late=True and task_reject_on_worker_lost=True so a killed task is redelivered, and make the task idempotent or resumable. Celery 5.5+ adds soft shutdown options that give tasks a window to clean up before a cold shutdown.

5. Docker and Compose

services:
  web:
    stop_grace_period: 45s
  worker:
    stop_grace_period: 10m

docker stop defaults to 10 seconds before SIGKILL, which cuts off longer requests and nearly every Celery task.

Verification check: start a slow request (for example a 20-second view) and trigger a deploy or docker stop: the request completes successfully; for Celery, start a long task and stop the worker: logs show Warm shutdown and the task finishes or is redelivered.

Explanation

Unix processes receive SIGTERM as a polite request to stop and SIGKILL as an unconditional kill. Production servers implement SIGTERM handlers that drain work, but the surrounding platform decides how long to wait before SIGKILL, and container conventions decide whether the server receives the signal at all. Graceful shutdown therefore requires the signal path, the drain logic, and the platform timeouts to be aligned, plus a pause so traffic routing catches up before the server stops accepting new connections.

Edge cases / notes

• WebSockets cannot finish gracefully; they close, and clients must reconnect.
• Long uploads may exceed graceful timeouts; upload directly to object storage instead.
• Beat should stop cleanly too; a killed Beat with a file scheduler may lose recent schedule state.
• Spot or preemptible nodes give limited warning; keep tasks short and idempotent.

Next steps

• Combine with rollouts in Zero-Downtime Django Deployment.
• Make entrypoints signal-safe with Write a Docker Entrypoint Script for Django.

FAQ

How does Gunicorn handle SIGTERM?

It stops accepting new connections and lets workers finish current requests for up to graceful_timeout seconds.

Why are requests dropped during deploys?

The signal may not reach Gunicorn, the platform kills too early, or traffic still arrives after shutdown starts. Add preStop and align timeouts.

What happens to Celery tasks on SIGTERM?

The worker performs a warm shutdown, finishing running tasks while taking no new ones.

What is Docker's default stop timeout?

Ten seconds before SIGKILL. Raise stop_grace_period for web and especially for workers.