Structured JSON Logging for Django

Problem statement

Plain-text logs such as 2026-10-02 10:15:03 WARNING orders payment failed for order 812 are fine to read one at a time and painful to search across thousands of lines: every query becomes a regular expression, and fields like order ID or user are buried in free text. Log platforms (Loki, CloudWatch Logs Insights, Datadog, Elasticsearch) work best with structured logs where each line is a JSON object with named fields. Django's logging is standard Python logging, so switching to JSON is a configuration change plus a few conventions.

1. Add a JSON formatter: python-json-logger for standard logging, or structlog for richer context binding.
2. Log to stdout and let the platform or agent collect it.
3. Include stable fields: timestamp, level, logger, message, request ID, user ID, path, status, duration.
4. Pass context with extra={...} (or structlog's bound context) instead of formatting it into the message.
5. Make Gunicorn access logs and Celery logs JSON too, so everything shares a schema.
6. Never log passwords, tokens, full card numbers, or raw request bodies.

Step-by-step solution

1. JSON with python-json-logger

pip install python-json-logger

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {
            "()": "pythonjsonlogger.json.JsonFormatter",
            "fmt": "%(asctime)s %(levelname)s %(name)s %(message)s",
            "rename_fields": {"asctime": "ts", "levelname": "level", "name": "logger"},
        },
    },
    "handlers": {"stdout": {"class": "logging.StreamHandler", "formatter": "json"}},
    "root": {"handlers": ["stdout"], "level": "INFO"},
    "loggers": {
        "django.request": {"level": "ERROR"},
        "django.db.backends": {"level": "WARNING"},
    },
}

Each record becomes one JSON line. (Older versions use pythonjsonlogger.jsonlogger.JsonFormatter; check your installed version.)

2. Log with fields, not formatted strings

log = logging.getLogger(__name__)
log.warning("payment failed", extra={"order_id": order.pk, "provider": "stripe", "error_code": err.code})

{"ts": "2026-10-02T10:15:03Z", "level": "WARNING", "logger": "orders.payments", "message": "payment failed", "order_id": 812, "provider": "stripe", "error_code": "card_declined"}

Now {logger="orders.payments"} | json | error_code="card_declined" in Loki, or filter error_code = "card_declined" in CloudWatch Insights, finds every case. Keep messages constant ("payment failed") so they group well, and put variable data in fields.

3. Add request context automatically

A small middleware stores request data in a context variable, and a logging filter adds it to every record:

import contextvars, logging, time, uuid
request_ctx = contextvars.ContextVar("request_ctx", default={})

class RequestContextMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
    def __call__(self, request):
        rid = request.headers.get("X-Request-ID") or uuid.uuid4().hex
        request_ctx.set({"request_id": rid, "path": request.path})
        start = time.monotonic()
        response = self.get_response(request)
        logging.getLogger("http").info("request", extra={
            "status": response.status_code,
            "duration_ms": round((time.monotonic() - start) * 1000, 1),
            "user_id": getattr(request.user, "pk", None),
        })
        response["X-Request-ID"] = rid
        return response

class ContextFilter(logging.Filter):
    def filter(self, record):
        for k, v in request_ctx.get().items():
            setattr(record, k, v)
        return True

Attach the filter to the handler ("filters": ["ctx"]). Context variables work for both sync and async views.

4. structlog alternative

structlog (with django-structlog) binds context once and adds it to all subsequent log calls, processes events through a pipeline (timestamps, levels, exceptions as structured fields), and renders JSON in production and pretty colours in development. It suits teams that log heavily.

5. Gunicorn and Celery

Use Gunicorn's logconfig_dict to route its access and error logs through the same JSON formatter, and set CELERY_WORKER_HIJACK_ROOT_LOGGER = False so Celery uses your configuration; add task name and ID with Celery signals or celery.app.log context.

Verification check: docker compose logs web | head (or journalctl -u gunicorn -o cat) shows one JSON object per line, your log platform parses fields automatically, and a query by request_id returns every line for one request.

Explanation

Python's logging system separates creating records (loggers), deciding where they go (handlers), and how they look (formatters). Switching the formatter to JSON changes the output without touching most application code, and extra attaches fields to individual records. Log platforms index those fields, so filtering and aggregation become exact and fast. Adding a request ID and other context to every record lets you reconstruct what happened in a single request across many log lines and services.

Edge cases / notes

• Exceptions are serialised as a string field with the traceback; structlog can render structured exception info.
• Log volume. JSON is larger than text; sample or drop noisy logs (health checks) at the source.
• Personal data. Logs are often retained longer and accessed more widely than databases; minimise personal data under GDPR.
• Local development. Use a human-readable formatter locally and JSON in production via settings.

Next steps

• Ship logs to a store with Ship Django Logs to Grafana Loki.
• Trace requests across services with Add Request IDs to Django Logs.

FAQ

How do I make Django logs JSON?

Configure a JSON formatter such as python-json-logger in LOGGING, or use structlog.

What fields should Django logs include?

Timestamp, level, logger, message, plus request ID, path, status, duration, and user ID where relevant.

Should I use structlog or python-json-logger?

python-json-logger is a minimal change; structlog offers context binding and processing pipelines for heavier logging.

Is it safe to log request data?

Log identifiers and metadata, not passwords, tokens, or full bodies.