Fix a Django Pod in CrashLoopBackOff

Problem statement

After a deploy, kubectl get pods shows the Django pods restarting again and again:

NAME                   READY   STATUS             RESTARTS      AGE
web-7c9f6d8b5-x2kqp    0/1     CrashLoopBackOff   6 (2m ago)    9m

CrashLoopBackOff is not an error in itself. It means the container keeps exiting and Kubernetes is waiting longer between each restart attempt. The reason the container exits is recorded in two places: the logs of the previous container instance and the pod's last termination state. For Django the usual causes are missing settings or secrets, a wrong start command, a liveness probe killing a slow-starting app, a memory limit, or a migration that fails at startup.

1. Read the crashed container's logs: kubectl logs <pod> --previous.
2. Check the termination reason: kubectl describe pod <pod> → Last State: Terminated, Reason: Error | OOMKilled, and the Events section.
3. Error with a traceback: fix settings, env vars, Secrets, or the command.
4. OOMKilled: raise the memory limit or reduce Gunicorn workers.
5. Probe failures in Events: add a startupProbe or longer initialDelaySeconds, and point probes at a cheap path.
6. Run migrations in a Job, not in the web container's startup.

Step-by-step solution

1. Get the previous container's logs

kubectl logs deploy/web --previous --tail=100
kubectl logs web-7c9f6d8b5-x2kqp --previous -c web

Without --previous, you often see an empty log from the container that just started. Look for Python tracebacks, ImproperlyConfigured, ModuleNotFoundError, or could not connect to server.

2. Read the termination state and events

kubectl describe pod web-7c9f6d8b5-x2kqp

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
...
Events:
  Warning  Unhealthy  Liveness probe failed: HTTP probe failed with statuscode: 400

• Exit code 1 or 3, Reason Error: the process failed (Python exception, Gunicorn could not load the app).
• Exit code 137, OOMKilled: the container exceeded its memory limit.
• Exit code 137 or 143 with probe events: the kubelet killed it after failed liveness probes.

3. Missing configuration and secrets

kubectl exec deploy/web -- env | sort        # if a container stays up long enough
kubectl get secret django-env -o jsonpath='{.data}' | head

Common mistakes: a Secret key name in envFrom/secretKeyRef that does not exist (the pod then fails with CreateContainerConfigError rather than crashing), a Secret value with a trailing newline from echo without -n, or DJANGO_SETTINGS_MODULE pointing at a module not in the image.

4. Probes that kill a working app

Probes request the pod IP, so Django sees a Host of the pod's IP and returns 400 unless it is allowed. Send a host header or allow it:

livenessProbe:
  httpGet:
    path: /health/
    port: 8000
    httpHeaders:
      - name: Host
        value: example.com
startupProbe:
  httpGet: { path: /health/, port: 8000, httpHeaders: [{ name: Host, value: example.com }] }
  failureThreshold: 30
  periodSeconds: 5

The startupProbe gives the app up to 150 seconds to start before liveness checks begin. Exempt the health path from SECURE_SSL_REDIRECT, or the probe receives a 301, which Kubernetes counts as success for 2xx-3xx, but a redirect to HTTPS on a plain HTTP port can still fail in some setups.

5. Memory limits

resources:
  requests: { memory: "384Mi", cpu: "250m" }
  limits:   { memory: "768Mi" }

Each Gunicorn worker is a full Django process. Four workers at 150 MB each plus the master need about 700 MB. Either lower --workers (and scale with replicas instead) or raise the limit.

6. Do not migrate in the web container

A command that runs migrate && gunicorn makes every replica try to migrate at startup. A failing migration crash-loops every pod, and concurrent migrations can deadlock. Use a Kubernetes Job or a Helm pre-upgrade hook; see Run Django Migrations as a Kubernetes Job.

Verification check:

kubectl rollout status deploy/web
kubectl get pods -l app=web
kubectl logs deploy/web --tail=20

Pods should be Running and 1/1 READY with a stable restart count.

Explanation

Kubernetes restarts containers that exit, using an exponential backoff (10 s, 20 s, 40 s, up to 5 minutes) to avoid hammering a broken workload. The status shown during the waiting period is CrashLoopBackOff. Each restart creates a new container instance, and kubectl logs shows the current one by default, which is often empty or still starting, while the useful error is in the instance that just died. The pod's lastState records why that instance ended, which separates application errors from kills by the kubelet for probes or memory.

Edge cases / notes

• ImagePullBackOff and CreateContainerConfigError are different: the container never started. Check image names, registry credentials, and referenced Secrets.
• Read-only root filesystem. Django or Gunicorn writing to a non-writable path (for example a temp directory) crashes on start; mount an emptyDir.
• Init containers that fail block the main container; check kubectl logs <pod> -c <init-container>.
• Rollback. kubectl rollout undo deploy/web returns to the last working version quickly.

Next steps

• Review the setup in Deploy Django on Kubernetes.
• Use the Django Kubernetes Production Checklist to prevent repeats.

FAQ

What does CrashLoopBackOff mean?

The container keeps exiting, and Kubernetes is waiting before restarting it again.

Why are the logs empty?

You are looking at the new container. Use kubectl logs --previous to see the one that crashed.

What does exit code 137 mean?

The container was killed with SIGKILL, usually for exceeding its memory limit (OOMKilled) or after failed liveness probes.

Should Django run migrations on pod startup?

No. Use a Job or pre-upgrade hook so migrations run once, separately from the web pods.