Reduce Django Log Volume and Logging Costs

Problem statement

Your log platform bill is approaching your hosting bill. Each request produces three access log lines (load balancer, Nginx, Gunicorn) plus Django's own, health checks from load balancers and Kubernetes add thousands of lines per minute, and a chatty library logs every Redis call at INFO. Meanwhile, the logs you need during incidents are hard to find in the noise. Reducing volume improves both cost and usefulness.

1. Measure: find the top log sources by line count in your platform (by logger, path, and service).
2. Drop health check, static file, and favicon requests from access logs.
3. Keep one access log per request (usually the load balancer or Nginx), and disable the others.
4. Raise noisy third-party loggers to WARNING; keep your app at INFO and errors everywhere.
5. Sample high-volume INFO events, and route verbose diagnostics to cheap, short-retention storage.

Step-by-step solution

1. Find the biggest sources

Group recent logs by service, logger name, and request path. Typical findings: GET /healthz/ from probes, Gunicorn and Nginx both logging every request, django.db.backends at DEBUG left on, botocore or urllib3 at INFO, and Celery worker heartbeats.

2. Drop health checks and static files from access logs

Nginx:

map $request_uri $loggable {
    ~^/healthz  0;
    ~^/static/  0;
    /favicon.ico 0;
    default     1;
}
access_log /var/log/nginx/access.log json if=$loggable;

Gunicorn's access log can be turned off when Nginx already logs requests (omit --access-logfile). If Gunicorn is the only access logger (for example on a PaaS), filter health checks in Django with a logging filter instead.

3. Set log levels per logger

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {"console": {"class": "logging.StreamHandler", "formatter": "json"}},
    "formatters": {"json": {"()": "pythonjsonlogger.json.JsonFormatter"}},
    "root": {"handlers": ["console"], "level": "WARNING"},
    "loggers": {
        "shop": {"level": "INFO", "handlers": ["console"], "propagate": False},
        "django.request": {"level": "ERROR"},
        "django.db.backends": {"level": "WARNING"},
        "botocore": {"level": "WARNING"},
        "urllib3": {"level": "WARNING"},
        "celery": {"level": "INFO"},
    },
}

Root at WARNING with your own package at INFO keeps your messages and silences dependency chatter.

4. Filter and sample in code

A logging filter can drop health check lines or sample routine events:

import logging, random

class SampleInfoFilter(logging.Filter):
    def __init__(self, rate=0.1):
        super().__init__()
        self.rate = rate
    def filter(self, record):
        return record.levelno > logging.INFO or random.random() < self.rate

Attach it to handlers for high-volume loggers only, never to error paths. Log aggregates (counts per minute) instead of one line per routine event when you only need totals; metrics are cheaper than logs for counting.

5. Route by value and retention

Send errors and app events to the main log platform with normal retention. Send verbose diagnostics (debug logs during incidents, full access logs for audits) to cheap object storage via your log shipper, with short or compliance-driven retention. Vector or Fluent Bit can route by level or logger before data reaches the paid platform.

Explanation

Log cost scales with ingested volume, and most volume comes from repetitive, low-information lines: probes, static assets, duplicated access logs, and libraries logging routine operations. Django's logging configuration controls which loggers emit what, proxies control access logs, and shippers can filter and route before ingestion. Keeping errors and meaningful application events while dropping repetition makes logs cheaper and easier to search when it matters.

Edge cases / notes

• Compliance. Some regulations require access logs; keep one complete copy in cheap storage rather than in the expensive search platform.
• Debugging incidents. Make levels adjustable at runtime (environment variable and restart, or a feature flag) so you can raise detail temporarily.
• Structured logs. JSON logs compress well and allow filtering on fields, which makes routing rules precise.

Next steps

• Set up logging properly with Set Up Django Logging in Production.
• Emit fields you can filter on with Django Structured JSON Logging.
• Route and filter at the edge with Ship Django Logs With Vector.

FAQ

What usually generates the most Django log volume?

Health check requests and duplicate access logs from several proxy layers.

Should I sample error logs?

No. Sample routine INFO events only; keep every warning and error.

Is Gunicorn's access log needed behind Nginx?

Usually not. One access log per request is enough.

How do I keep logs for compliance cheaply?

Ship full access logs to object storage with lifecycle rules, and keep only searchable subsets in the log platform.