Add OpenTelemetry Tracing to Django

Problem statement

Metrics tell you that the checkout endpoint's p95 latency doubled. Logs show individual requests. Neither shows where the time went inside a slow request: a particular SQL query, a Redis call, an external payment API, or a Celery task triggered afterwards. Distributed tracing records each operation in a request as a span with timing, attributes, and parent-child relationships, and follows the same trace across services.

OpenTelemetry (OTel) is the vendor-neutral standard for traces, metrics, and logs, with instrumentation libraries for Django and common Python packages. Adding it to a Django deployment involves a few production-specific decisions:

• how to initialize tracing in Gunicorn workers without deadlocks caused by forking
• which libraries to instrument
• how much to sample, so tracing does not add cost or overhead out of proportion to its value
• where to send data (an OpenTelemetry Collector, then your tracing backend)
• how to avoid recording sensitive data in span attributes

Quick answer

1. Install opentelemetry-sdk, opentelemetry-exporter-otlp, and instrumentation packages for Django, psycopg, Redis, requests, and Celery.
2. Initialize the tracer provider and instrumentations in each Gunicorn worker (post_fork hook), not in the master.
3. Set the service name, environment, and version as resource attributes.
4. Export spans with OTLP to an OpenTelemetry Collector running near the app.
5. Use parent-based ratio sampling (for example 10 percent) and exclude health checks.
6. Add custom spans around important business operations.
7. Verify traces appear with database and HTTP child spans, and correlate them with logs.

Step-by-step solution

1. Install packages

opentelemetry-sdk
opentelemetry-exporter-otlp
opentelemetry-instrumentation-django
opentelemetry-instrumentation-psycopg
opentelemetry-instrumentation-redis
opentelemetry-instrumentation-requests
opentelemetry-instrumentation-celery

Pin versions that match each other: OpenTelemetry Python releases the SDK and instrumentation packages in coordinated versions. If you use psycopg2, install opentelemetry-instrumentation-psycopg2 instead of the psycopg 3 package.

2. Write an initialization module

# config/tracing.py
import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.django import DjangoInstrumentor
from opentelemetry.instrumentation.psycopg import PsycopgInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased


def init_tracing():
    if os.environ.get("TRACING_ENABLED", "0") != "1":
        return

    resource = Resource.create({
        "service.name": os.environ.get("OTEL_SERVICE_NAME", "django-web"),
        "deployment.environment": os.environ.get("APP_ENV", "production"),
        "service.version": os.environ.get("APP_RELEASE", "unknown"),
    })

    ratio = float(os.environ.get("TRACING_SAMPLE_RATIO", "0.1"))
    provider = TracerProvider(resource=resource, sampler=ParentBased(TraceIdRatioBased(ratio)))
    provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint=os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:4317"))
        )
    )
    trace.set_tracer_provider(provider)

    DjangoInstrumentor().instrument()
    PsycopgInstrumentor().instrument()
    RedisInstrumentor().instrument()
    RequestsInstrumentor().instrument()

• Resource attributes identify the service, environment, and release in the tracing backend, which makes comparing releases possible.
• ParentBased(TraceIdRatioBased(0.1)) keeps 10 percent of new traces and follows the sampling decision of incoming requests that already carry trace context, so traces stay complete across services.
• Excluded URLs: set OTEL_PYTHON_DJANGO_EXCLUDED_URLS=healthz,readyz,metrics in the service environment so health checks and metrics scrapes do not produce useless traces. The value is a comma-separated list of patterns matched against request URLs.

The Django instrumentation works through middleware it inserts; initializing it before Django handles requests is required.

3. Initialize after Gunicorn forks

BatchSpanProcessor uses a background thread to export spans. If it starts in the Gunicorn master before workers fork, children can inherit locks held by that thread and deadlock. Initialize in each worker instead:

# gunicorn.conf.py
wsgi_app = "config.wsgi:application"
bind = "unix:/run/myapp/gunicorn.sock"
workers = 4


def post_fork(server, worker):
    from config.tracing import init_tracing

    init_tracing()

Do not also run the application under the opentelemetry-instrument wrapper command when initializing manually, or instrumentation is applied twice. With --preload, make sure nothing in the preloaded import path starts span processors.

4. Run an OpenTelemetry Collector

Send spans to a local Collector rather than directly to a vendor. The Collector batches, retries, filters sensitive attributes, and forwards to one or more backends:

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317

processors:
  batch: {}
  attributes/scrub:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: db.statement
        action: hash

exporters:
  otlphttp:
    endpoint: https://otlp.tracing-vendor.example.com
    headers:
      api-key: ${env:TRACING_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/scrub, batch]
      exporters: [otlphttp]

Bind the receiver to localhost (or a private network) so only your services can send data. Adjust attribute names to the semantic conventions used by your instrumentation versions; they have evolved over time.

5. Instrument Celery

Tracing across the web request and the background task that it enqueues shows the full picture. Initialize instrumentation in each Celery worker process after it starts:

# config/celery.py
from celery import Celery
from celery.signals import worker_process_init

app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()


@worker_process_init.connect(weak=False)
def init_celery_tracing(*args, **kwargs):
    from opentelemetry.instrumentation.celery import CeleryInstrumentor

    from config.tracing import init_tracing

    init_tracing()
    CeleryInstrumentor().instrument()

worker_process_init runs in each prefork child, the Celery equivalent of Gunicorn's post_fork. Set OTEL_SERVICE_NAME=django-worker for workers so web and worker spans are distinguishable. Trace context propagates through task message headers when both the producer (web process) and workers are instrumented.

6. Add custom spans

Automatic instrumentation shows framework and library calls. Add spans for business operations whose timing matters:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)


def place_order(cart, user):
    with tracer.start_as_current_span("checkout.place_order") as span:
        span.set_attribute("cart.item_count", len(cart.items))
        with tracer.start_as_current_span("checkout.reserve_stock"):
            reserve_stock(cart)
        with tracer.start_as_current_span("checkout.charge_payment"):
            charge = charge_payment(cart, user)
        span.set_attribute("payment.provider", charge.provider)
        return create_order(cart, user, charge)

Never put personal data, secrets, or full request bodies into span attributes. Use counts, identifiers that are safe to share with your tracing provider, and enumerated values.

7. Correlate logs with traces

Include trace and span IDs in log records so you can jump from a log line to its trace:

import logging

from opentelemetry import trace


class TraceContextFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        record.trace_id = format(ctx.trace_id, "032x") if ctx.is_valid else "-"
        record.span_id = format(ctx.span_id, "016x") if ctx.is_valid else "-"
        return True

Add the filter to your handlers and trace_id=%(trace_id)s to the log format.

8. Configure sampling and overhead

• Start with 5 to 20 percent sampling for web traffic, and 100 percent in staging.
• Instrumentation adds small per-span costs; the database instrumentation creates one span per query, so N+1 endpoints produce very large traces. That is useful for finding them, but consider tail-based sampling in the Collector (keep slow or failed traces, sample the rest) for busy services.
• Exporting is asynchronous and batched; if the Collector is unavailable, spans are dropped rather than blocking requests.

9. Verify

In staging with TRACING_SAMPLE_RATIO=1.0:

curl -s https://staging.example.com/orders/ > /dev/null

In the tracing backend, the trace should show:

• a root span for the Django request with route and status attributes
• child spans for each SQL query with durations
• spans for Redis calls and outgoing HTTP requests
• a linked span for a Celery task enqueued by the request

Also confirm no worker deadlocks after deploys (workers keep serving requests) and that health checks do not produce traces.

Verification check:

• traces appear with correct service name, environment, and release
• database and external call spans are nested under request spans
• Celery task spans continue the web request's trace
• sensitive attributes are absent or scrubbed
• request latency is not noticeably higher with tracing enabled at the chosen sample rate

Explanation

A trace is the tree of operations performed for one request, possibly across several processes and services. OpenTelemetry's instrumentation libraries create spans for Django requests, database queries, cache calls, and HTTP clients automatically, while custom spans capture business steps. The production-specific work is initialization (per worker, after forking), controlling volume (sampling and exclusions), and data hygiene (Collector processing), so tracing remains cheap, safe, and useful.

Edge cases / notes

ASGI servers

Uvicorn workers also fork when started with multiple workers. Initialize tracing in each worker process, for example at application startup inside the ASGI app module rather than in a parent process.

Auto-instrumentation wrapper

opentelemetry-instrument can configure everything from environment variables. It is convenient for single-process servers; with preforking servers, follow the documented fork-safety guidance for your version or use manual initialization as shown.

Existing error trackers

Sentry and other tools also offer performance tracing. Running two tracing systems doubles overhead; choose one as the primary source of traces.

Using traces during an incident

Traces earn their cost when something is slow or failing. A practical workflow:

1. Start from the symptom: filter traces by service, route, and status code, or by latency above a threshold.
2. Compare a slow trace with a fast one for the same route. The difference usually stands out: extra database spans (an N+1 pattern that only appears with certain data), a slow external call, or a long gap with no spans (time spent in Python code, which is where a profiler helps).
3. Check the release attribute: if slow traces all carry the new release version, the regression came with a deploy.
4. Follow the trace across services: a request that enqueues a Celery task shows whether the delay is in the request or in the background processing.
5. Confirm with metrics: traces show individual requests; metrics confirm whether the pattern is widespread.

Adding a few well-chosen attributes makes this filtering possible: tenant or plan type, feature flag state, and cache hit or miss for expensive lookups. Keep them low-cardinality and free of personal data.

Cost control

Tracing costs scale with span volume, which scales with database queries per request. Practical controls:

• Head sampling at the SDK level (the ratio in the configuration above) for a simple, predictable reduction.
• Tail sampling in the Collector, keeping all traces that contain errors or exceed a latency threshold, plus a small share of the rest. This keeps the interesting traces at a fraction of the volume.
• Span limits for endpoints with unavoidable query volume, or disabling database instrumentation for specific noisy paths.
• Shorter retention for routine traces, longer for error traces if your backend supports it.

Review span volume monthly as traffic grows, the same way you review log volume.

Common mistakes

• Initializing the span processor in the Gunicorn master and forking workers afterwards.
• Tracing 100 percent of production traffic on a busy site without tail sampling.
• Recording full SQL with parameters, headers, or user data in span attributes.
• Instrumenting web processes but not Celery workers, breaking traces at task boundaries.

Next steps

• Combine traces with metrics in Monitor Django with Prometheus and Grafana.
• Investigate CPU-heavy code paths with Profile Slow Django Endpoints in Production with py-spy.
• Capture exceptions with Set Up Sentry Error Monitoring for Django.
• Structure logs for correlation with Django Logging Setup for Production.

FAQ

How do I add OpenTelemetry to a Django app running on Gunicorn?

Install the SDK, OTLP exporter, and instrumentation packages, then initialize the tracer provider and instrumentors in Gunicorn's post_fork hook so each worker has its own span processor.

Why does OpenTelemetry cause Gunicorn workers to hang?

A BatchSpanProcessor started in the master process owns a background thread and locks. Forked workers can inherit a held lock and deadlock. Initialize tracing after forking.

What sampling rate should I use for Django tracing?

Start around 5 to 20 percent for production web traffic with parent-based sampling, and use 100 percent in staging. Tail-based sampling in the Collector keeps slow and failed traces while reducing volume.

Can OpenTelemetry trace Celery tasks from Django requests?

Yes. Instrument both the web process and Celery workers; trace context travels in task message headers, so task spans join the originating request's trace.