Fix Nginx "no live upstreams" in Front of Django

Problem statement

After a deploy or a burst of errors, every request returns 502 for a few seconds and the Nginx error log fills with:

no live upstreams while connecting to upstream, client: 198.51.100.4,
server: example.com, request: "GET / HTTP/1.1", upstream: "http://django_app/"

Nginx did not even try to connect. Its passive health tracking had marked every server in the upstream block as unavailable, so there was nowhere to send the request. The backends may already be fine again; Nginx is waiting out fail_timeout before it retries them. The real question is what made every backend fail at once.

1. Look at the errors before the first "no live upstreams" line; they show why servers were marked down.
2. Fix the underlying failures: all app servers restarting together, timeouts, or connection refused.
3. Do not count application timeouts as server failures on slow endpoints: tune proxy_next_upstream and max_fails.
4. With a single backend, max_fails marking it down only adds downtime; use max_fails=0 for single-server upstreams.
5. Restart app servers one at a time during deploys.

Step-by-step solution

1. Find the first failures

sudo grep -B 20 -m1 "no live upstreams" /var/log/nginx/error.log

Typical predecessors:

• connect() failed (111: Connection refused) — the app server was not listening (restart in progress, crashed).
• upstream timed out (110: Connection timed out) — requests slower than proxy_read_timeout.
• upstream prematurely closed connection — workers killed mid-request.

Each of these counts as a failure for the server it happened on.

2. Understand max_fails and fail_timeout

upstream django_app {
    server 10.0.0.11:8000 max_fails=3 fail_timeout=10s;
    server 10.0.0.12:8000 max_fails=3 fail_timeout=10s;
}

The defaults are max_fails=1 and fail_timeout=10s: one failed attempt within 10 seconds marks the server down for 10 seconds. With two servers, two unlucky requests can take the whole upstream out. Raising max_fails to 3 makes Nginx tolerate occasional errors.

3. Decide what counts as a failure

proxy_next_upstream controls which errors make Nginx retry another server and count a failure. The default is error timeout. For Django, keep retries for connection errors but avoid retrying slow, non-idempotent requests:

location / {
    proxy_pass http://django_app;
    proxy_next_upstream error timeout http_502 http_503;
    proxy_next_upstream_tries 2;
    proxy_next_upstream_timeout 5s;
    proxy_connect_timeout 2s;
}

Nginx does not retry non-idempotent requests (POST, PATCH) after the request was sent unless you add non_idempotent, which you should not do for Django forms or payments.

A short proxy_connect_timeout makes a dead backend fail fast and lets Nginx try the next one, while a long proxy_read_timeout still allows slow pages.

4. Single backend upstreams

With one server, marking it down is pointless: there is no alternative, and Nginx adds fail_timeout seconds of guaranteed 502s after each blip. Disable failure tracking:

upstream django_app {
    server unix:/run/gunicorn.sock max_fails=0;
}

or use proxy_pass http://unix:/run/gunicorn.sock; directly without an upstream block.

5. Roll deploys across servers

Restart backends one at a time and wait for each to pass a health check before the next. With Gunicorn, prefer systemctl reload (HUP) on each host, which keeps the socket open, over a full restart. See Zero-Downtime Django Deployment.

Verification check: during the next deploy, tail the error log: sudo tail -f /var/log/nginx/error.log | grep -E "no live|refused". With rolling restarts and tuned settings, no "no live upstreams" lines should appear.

Explanation

Open-source Nginx uses passive health checks: it learns that a server is unhealthy only from failed real requests. Once max_fails failures happen within fail_timeout, the server is skipped for fail_timeout. When every server is skipped, Nginx has nothing to try and logs "no live upstreams". Active health checks that probe a /health/ URL are available in NGINX Plus, HAProxy, Traefik, and cloud load balancers, which mark servers down and back up based on probes instead of user traffic.

Edge cases / notes

• Kubernetes. With ingress-nginx, Service endpoints come and go via readiness probes, so this message usually means no pod was Ready. Check readiness probe failures.
• DNS names in upstream. Open-source Nginx resolves upstream hostnames at startup only. If container IPs change, Nginx keeps sending to old addresses; use resolver with a variable in proxy_pass or reload Nginx.
• backup servers marked with backup receive traffic only when all primary servers are down, which can serve a maintenance page.

Next steps

• Build a proper multi-server setup with Scale Django Horizontally Behind a Load Balancer.
• Work through other 502 causes in Debug Django 502 Bad Gateway.

FAQ

Why does Nginx return 502 even though Gunicorn is running again?

The server is still within its fail_timeout window. Nginx waits that long before trying it again.

Can I disable failure tracking?

Yes, with max_fails=0 on each server. That is the right choice for a single backend.

Does Nginx retry POST requests on another server?

Not after the request was sent, unless non_idempotent is set. Leave it off for Django.

Does open-source Nginx support active health checks?

No. Use NGINX Plus, HAProxy, or a cloud load balancer for active checks.