Add Request IDs to Django Logs

Problem statement

A user reports "the checkout failed at 10:15". The logs around that time contain hundreds of interleaved lines from many workers and requests, and there is no way to tell which lines belong to that one checkout, let alone which Celery task it triggered. A request ID, a unique identifier assigned when the request enters your system and attached to every log line, error report, and downstream call, turns that search into a single filter. It is the lightweight first step towards full distributed tracing.

1. Generate the ID at the edge: Nginx proxy_set_header X-Request-ID $request_id; (or reuse the load balancer's trace header).
2. In Django, read it in middleware (or generate one), store it in a contextvars.ContextVar, and return it in the response header.
3. Add a logging filter that copies it onto every log record.
4. Tag Sentry events with it so error reports link to logs.
5. Pass it to Celery tasks via task headers and restore it in the worker.
6. Forward it on outbound HTTP calls to your other services.

Step-by-step solution

1. Set it at the proxy

log_format main_rid '$remote_addr "$request" $status $request_time rid=$request_id';
access_log /var/log/nginx/access.log main_rid;

location / {
    proxy_pass http://unix:/run/gunicorn/gunicorn.sock;
    proxy_set_header X-Request-ID $request_id;
    include proxy_params;
}

$request_id is a random 32-character hex value Nginx generates per request. Logging it in the access log connects proxy lines to app lines. Behind an AWS ALB, X-Amzn-Trace-Id serves a similar role; Cloudflare provides CF-Ray.

2. Store it in Django

# core/request_id.py
import contextvars, re, uuid

request_id_var = contextvars.ContextVar("request_id", default="-")
_VALID = re.compile(r"^[A-Za-z0-9\-]{8,64}$")

class RequestIDMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        incoming = request.headers.get("X-Request-ID", "")
        rid = incoming if _VALID.match(incoming) else uuid.uuid4().hex
        token = request_id_var.set(rid)
        try:
            response = self.get_response(request)
        finally:
            request_id_var.reset(token)
        response["X-Request-ID"] = rid
        return response

Validate incoming IDs (length and characters) so clients cannot inject junk into logs. Put the middleware near the top of MIDDLEWARE so it wraps everything else.

3. Add it to every log line

class RequestIDFilter(logging.Filter):
    def filter(self, record):
        record.request_id = request_id_var.get()
        return True

LOGGING["filters"] = {"request_id": {"()": "core.request_id.RequestIDFilter"}}
LOGGING["handlers"]["stdout"]["filters"] = ["request_id"]
# text format example: "%(asctime)s %(levelname)s [%(request_id)s] %(name)s %(message)s"

With JSON logging, request_id becomes a field automatically.

4. Sentry and error pages

import sentry_sdk
sentry_sdk.set_tag("request_id", rid)    # inside the middleware after setting the var

Render the ID on your 500.html ("Reference: " via a context processor) so users can quote it.

5. Celery tasks

from celery.signals import before_task_publish, task_prerun

@before_task_publish.connect
def add_request_id(headers=None, **kwargs):
    headers["request_id"] = request_id_var.get()

@task_prerun.connect
def restore_request_id(task=None, **kwargs):
    rid = getattr(task.request, "request_id", None) or uuid.uuid4().hex
    request_id_var.set(rid)

Now logs from the task that a request queued carry the same ID.

6. Outbound calls

Send X-Request-ID on HTTP calls to your other services (a requests.Session hook or httpx event hook) so their logs join the same trail. For full timing breakdowns across services, adopt OpenTelemetry, which propagates trace context in a standard way.

Verification check: request a page, read X-Request-ID from the response, and search your logs for it: the Nginx access line, Django log lines, and any Celery task lines should all appear.

Explanation

Web servers handle many requests concurrently, so log lines from different requests interleave. A request ID is a correlation key: generated once at the boundary, carried through the request's processing, and attached to everything emitted along the way. Context variables provide per-request storage that works across threads and async tasks, and logging filters inject the value without changing every log call. Propagating it through task queues and HTTP headers extends the correlation beyond a single process.

Edge cases / notes

• Trust boundary. Accept client-supplied IDs only after validation, or always generate your own and log the incoming one separately.
• OpenTelemetry uses the W3C traceparent header; if you adopt it, log the trace ID instead of or alongside a custom request ID.
• Streaming responses finish after the middleware returns; reset the context carefully or use the response's close hook.
• Management commands can set a run ID in the same contextvar for consistent logs.

Next steps

• Make logs queryable with Structured JSON Logging for Django.
• Go further with Django OpenTelemetry Tracing.

FAQ

What is a request ID?

A unique identifier assigned to each incoming request and attached to all logs and errors it produces.

Where should the request ID be generated?

At the edge (Nginx or the load balancer), with Django generating one if none is present.

How do I add the request ID to Django logs?

Store it in a contextvar in middleware and add a logging filter that copies it onto each record.

Can the request ID follow Celery tasks?

Yes. Add it to task headers when publishing and restore it in task_prerun.