Profile Slow Django Endpoints in Production with py-spy

Problem statement

An endpoint is slow in production but fast in development, or Gunicorn workers occasionally hang until they time out. Reproducing the problem locally fails because it depends on production data volumes, concurrency, or a specific external service. Adding logging and redeploying for each hypothesis is slow, and full Python profilers (cProfile) add heavy overhead and require code changes.

py-spy is a sampling profiler for Python that attaches to a running process from outside. It reads the process's memory to capture Python stack traces many times per second, without modifying code or restarting the application, and with low overhead. For Django on Gunicorn it can:

• show which functions consume time in a running worker (top)
• record a flame graph of where time goes over a period (record)
• print the current stack of a stuck worker (dump), which answers "what is it waiting on?"

This guide uses py-spy safely on production Django workers and interprets the results.

Quick answer

1. Install py-spy on the server (or in the container) with pip install py-spy.
2. Find Gunicorn worker PIDs with ps or pgrep.
3. For a hung worker, run sudo py-spy dump --pid <PID> to see exactly where it is blocked.
4. For slowness under load, run sudo py-spy record -o profile.svg --pid <PID> --duration 60 while the slow traffic happens.
5. Use --idle to include time spent waiting (I/O), and --subprocesses or multiple PIDs to cover several workers.
6. Read the flame graph from the bottom (entry points) to the widest frames (where time accumulates).
7. Fix the hot spot, deploy, and profile again to confirm.

Step-by-step solution

1. Install py-spy

py-spy is a standalone binary distributed through PyPI; it does not need to be installed in the application's virtualenv:

sudo python3 -m pip install py-spy
# or with pipx:
pipx install py-spy
py-spy --version

Attaching to another process requires permission to read its memory. On most Linux systems, run it with sudo, or as the same user if kernel.yama.ptrace_scope allows it.

2. Containers need ptrace permission

In Docker, reading another process's memory requires the SYS_PTRACE capability:

docker run --cap-add SYS_PTRACE ...

or in Compose:

services:
  web:
    cap_add:
      - SYS_PTRACE

Alternatively, run py-spy from the host against the container's process ID as seen from the host. In Kubernetes, use an ephemeral debug container sharing the process namespace, subject to your cluster's security policies.

3. Find the right process

ps -o pid,ppid,etime,pcpu,rss,cmd -C gunicorn --sort=-pcpu

The master has PPID 1 (or the systemd service); workers are its children. The master does not handle requests, so profile workers. If one worker shows high CPU or long elapsed time on a request, start there.

4. Dump a stuck worker

When a worker seems hung (requests timing out, WORKER TIMEOUT messages), capture its current stack:

sudo py-spy dump --pid 23114

Example output:

Thread 23114 (idle): "MainThread"
    read (ssl.py:1134)
    recv_into (ssl.py:1307)
    readinto (socket.py:720)
    _read_status (http/client.py:292)
    begin (http/client.py:331)
    getresponse (http/client.py:1428)
    _make_request (urllib3/connectionpool.py:...)
    send (requests/adapters.py:...)
    get (requests/api.py:...)
    fetch_exchange_rates (billing/rates.py:42)
    invoice_detail (billing/views.py:88)

This worker is blocked reading an HTTPS response in fetch_exchange_rates, called from a view. The fix is clear: add a timeout to the requests call and cache or precompute exchange rates.

--locals also prints local variables of each frame, which can include sensitive data such as tokens or personal information. Use it only when necessary and do not paste the output into shared channels.

Take several dumps a few seconds apart; if the stack is identical each time, the worker is truly stuck there.

5. See live hot functions

sudo py-spy top --pid 23114

top shows a continuously updating table of functions by sampled time, similar to the Unix top command. It helps confirm whether a worker is CPU-busy in Python code or mostly idle.

6. Record a flame graph

While the slow endpoint is being exercised (by real traffic, or by replaying requests against a single worker in staging):

sudo py-spy record \
  --pid 23114 \
  --duration 60 \
  --rate 100 \
  --idle \
  --output /tmp/invoice-profile.svg

• --duration limits the recording time.
• --rate sets samples per second (default 100); lower rates reduce overhead further.
• --idle includes samples where the thread is waiting, which matters for Django: time waiting on the database or HTTP calls is often most of a slow request.

Copy the SVG to your machine and open it in a browser. Frames are interactive: click to zoom.

To profile all workers of one Gunicorn instance, profile the master with --subprocesses:

sudo py-spy record --pid <MASTER_PID> --subprocesses --duration 60 --idle -o /tmp/all-workers.svg

7. Read the flame graph

• The x-axis is the share of samples, not time order. Wider frames used more samples.
• The y-axis is the stack: callers below, callees above.
• Look for wide frames in your own code (myapp/...), then read upward to see what they call.

Common Django findings:

8. Profile safely in production

• Keep recordings short (30 to 120 seconds) and at the default or lower sampling rate.
• Profile one worker at a time on a busy server; overhead is low but not zero.
• Remember that dump --locals exposes variable contents.
• Delete profile files when done if they contain function arguments or paths you consider sensitive.
• Prefer reproducing in staging with production-like data when the issue is easy to trigger there.

9. Complementary tools

• Database side: pg_stat_statements and EXPLAIN ANALYZE show which queries are slow once py-spy points to database time.
• Tracing: OpenTelemetry traces show per-request timing across services continuously; py-spy is better for ad hoc deep dives into CPU and stuck code.
• Memory: py-spy samples CPU and wall time, not memory allocations. Use tracemalloc or memray for memory growth.

10. Verify the fix

After deploying a change, repeat the measurement:

curl -s -o /dev/null -w "%{time_total}\n" https://www.example.com/invoices/1042/
sudo py-spy record --pid <PID> --duration 60 --idle -o /tmp/invoice-profile-after.svg

Verification check:

• the previously wide frames shrink or disappear in the new flame graph
• p95 latency for the endpoint drops in access logs or APM
• no more WORKER TIMEOUT entries related to the endpoint
• dumps of busy workers no longer show the same blocking stack

Explanation

Sampling profilers periodically record the call stack of a running process. Functions that appear in many samples are where the process spends its time, whether computing or waiting. Because py-spy reads the target process's memory from outside, it works on production workers without code changes or restarts, which makes it well suited to problems that only appear with production data and traffic. For Django, including idle time is important, since slow requests often spend most of their time waiting on the database or external services.

Edge cases / notes

Async code

For ASGI deployments, py-spy shows thread stacks, including the event loop thread. Coroutine-level views are harder to read; --native and async-aware tools can help.

Native extensions

Time spent inside C extensions (for example image processing or cryptography) appears under the Python frame that called them. --native includes native frames when you need more detail, at higher overhead.

Short requests

For very fast endpoints called frequently, a single 60-second recording aggregates thousands of requests, which is exactly what you want to find cumulative costs.

Common mistakes

• Profiling the Gunicorn master instead of workers.
• Recording without --idle, which hides database and network waiting.
• Using dump --locals casually on production processes handling personal data.
• Optimizing code that appears in the profile but is not on the slow endpoint's path.

Next steps

• Fix query-heavy profiles with Find and Fix N+1 Queries in Django.
• If requests time out at the proxy, see Fix Django 504 Gateway Timeout Behind Nginx.
• Add continuous request timing with Add OpenTelemetry Tracing to Django.
• Understand worker kills in Gunicorn Worker Timeout in Django: Root Causes and Fixes.

FAQ

Can I profile a running Django app without restarting it?

Yes. py-spy attaches to a running Gunicorn worker by PID and samples its stacks from outside, with no code changes or restart.

Is py-spy safe to use in production?

It is designed for production use with low overhead. Keep recordings short, profile one worker at a time on busy servers, and avoid dumping local variables on processes that handle sensitive data.

How do I see why a Gunicorn worker is stuck?

Run sudo py-spy dump --pid <worker PID> several times. A repeated identical stack shows the exact line where the worker is blocked, such as a network call without a timeout.

Why do I need --idle when profiling Django?

Without it, py-spy ignores samples where the thread is waiting. Django requests often spend most time waiting on databases and APIs, so --idle reveals those costs.