Performance & scaling for Django in production

Set it up

Guides that take this from nothing to running in production.

• Set Up PgBouncer for Django

  Put PgBouncer between Django and PostgreSQL: transaction pooling, SCRAM auth, pool sizing, compatible Django settings, and monitoring.

• Load Balance Django With HAProxy

  Put HAProxy in front of several Django app servers: active health checks on a /health/ endpoint, TLS termination, X-Forwarded-Proto, connection limits per server, rolling deploys with drain, and stick tables for basic rate limiting.

• Serve a Django Site Through Fastly

  Put Fastly in front of Django: configure the origin and host header, cache anonymous pages with Surrogate-Control and surrogate keys, bypass logged-in users, purge by key from Django signals, and set shielding, TLS, and compression for static files.

Run it well

Tuning, reliability, and operations once it works.

• How to Load Test a Django App Before Release

  Pre-release load testing answers a practical deployment question: will this Django release survive real traffic on your actual production stack.

• Find and Fix N+1 Queries in Django

  Find and fix N+1 queries in Django: detect them in tests and logs, use select_related, prefetch_related, Prefetch objects, and annotations correctly.

• Enable Gzip and Brotli Compression for Django with Nginx

  Compress Django responses and static files correctly: Nginx gzip settings, precompressed static assets, Brotli support, BREACH caveats, and checks.

• Scale Django Horizontally Behind a Load Balancer

  Prepare Django to run on multiple servers behind a load balancer: shared sessions, media, caches, migrations, scheduled jobs, health checks, and deploys.

• Size a Server for a Django App: Capacity Planning

  Estimate CPU, memory, and database capacity for Django: measure per-request costs, calculate worker counts, plan headroom, load test, and scale.

• Profile Slow Django Endpoints in Production with py-spy

  Find why Django endpoints are slow in production: sample live Gunicorn workers with py-spy, record flame graphs, and dump stuck workers safely.

• Find Slow Django Queries with pg_stat_statements

  Enable pg_stat_statements for a Django app, rank queries by total time, trace them back to ORM code, confirm plans with EXPLAIN ANALYZE, and fix them with indexes and better querysets.

• Reduce Django Hosting Costs Without Losing Reliability

  Cut the cost of running Django in production: measure before resizing, right-size servers and databases, trim idle environments, control egress, NAT, logs, and backups, and commit only to steady usage.

• Set Cache-Control Headers in Django for CDNs and Browsers

  Control how browsers and CDNs cache Django responses: cache_control and never_cache decorators, public versus private, max-age versus s-maxage, Vary headers from sessions and cookies, ETags, and safe defaults for logged-in pages.

• Cache Django Pages at Cloudflare's Edge

  Cache Django HTML at Cloudflare’s edge safely: Cache Rules for anonymous pages, bypassing cache when session or CSRF cookies exist, respecting origin Cache-Control, purging on deploy or content change, and avoiding cached private pages.

• Reduce Django Startup and Import Time

  Make Django processes start faster: measure import time with python -X importtime, defer heavy imports, avoid database queries at import, trim INSTALLED_APPS, and why startup matters for serverless, autoscaling, deploys, and management commands.

• Export Large CSV Files From Django Without Timeouts

  Export large datasets from Django as CSV without timeouts or memory spikes: StreamingHttpResponse with iterator(), values_list to skip model instances, Nginx buffering, background exports to object storage for very large files, and Excel compatibility.

• Use ETags and Conditional Requests in Django

  Cut bandwidth and server work with HTTP conditional requests in Django: ConditionalGetMiddleware, the condition, etag, and last_modified decorators, cheap ETag functions, 304 Not Modified responses for APIs, and how ETags interact with CDNs and compression.

• Speed Up Django Template Rendering With the Cached Loader

  Make Django templates render faster in production: confirm the cached template loader is active, avoid disabling it with custom loaders or APP_DIRS mistakes, reduce expensive template work, use fragment caching, and measure template time per view.

• Reduce Django Memory Usage per Worker

  Lower the memory footprint of Django processes: measure RSS per Gunicorn and Celery worker, trim imports and INSTALLED_APPS, stream large querysets and responses, recycle workers with max-requests, try jemalloc, and size worker counts to fit the server.

• Deploy Django Across Multiple Regions

  Plan a multi-region Django deployment: decide between active-passive and active-active, route users with latency or geo DNS, place PostgreSQL primaries and read replicas, handle sessions, caches, Celery, and media per region, and test regional failover.

• Continuous Profiling for Django With Pyroscope

  Add always-on continuous profiling to Django and Celery with Grafana Pyroscope: install the Python SDK or use eBPF, tag profiles by view and task, read flame graphs to find hot code paths, compare releases, and keep overhead low in production.

• How to Optimize Gunicorn Workers for Django

  If you want to optimize Gunicorn workers for Django, the main production risk is getting concurrency wrong.

• Django Caching Strategy for Production

  Choose what to cache in a Django app: HTTP caching, per-view and fragment caches, low-level caching, invalidation, stampede protection, and hit rates.

• Configure Persistent Database Connections in Django

  Use CONN_MAX_AGE and CONN_HEALTH_CHECKS correctly: when persistent connections help, connection math, idle timeouts, ASGI caveats, and measuring gains.

• Use Django's Built-In PostgreSQL Connection Pool

  Enable Django's native psycopg 3 connection pool: sizing per process, CONN_MAX_AGE rules, health checks, Gunicorn and ASGI behavior, and PgBouncer.

• Build Smaller, Faster Django Docker Images

  Shrink and speed up Django Docker images: slim base images, multi-stage builds, layer ordering, BuildKit cache mounts, .dockerignore, and measurement.

• Route Django Reads to PostgreSQL Read Replicas

  Add PostgreSQL read replicas to Django: multi-database settings, routers, replication lag and read-your-writes, transactions, migrations, and tests.

• Tune Celery Worker Concurrency and Prefetch for Django

  Tune Celery workers for Django workloads: pool types, concurrency, prefetch multiplier, acks_late, time and memory limits, and dedicated queues.

• Long-Lived Caching for Django Static Files

  Cache Django static files for a year without stale assets: manifest hashing, immutable Cache-Control headers in Nginx and WhiteNoise, CDNs, and deploy order.

• Enable HTTP/2 and HTTP/3 for Django with Nginx

  Serve a Django site over HTTP/2 and HTTP/3 (QUIC) with Nginx: version-specific directives, Alt-Svc, UDP 443, keeping HTTP/1.1 to Gunicorn, verifying with curl, and what actually gets faster.

• Set PostgreSQL Statement Timeouts for Django

  Stop runaway queries from exhausting Django workers: set statement_timeout per connection or role, give migrations and Celery different limits, handle QueryCanceled errors, and work with PgBouncer.

• Tune PostgreSQL Autovacuum for Django

  Keep Django tables fast and small with autovacuum tuning: find dead tuples and bloat, set per-table thresholds for hot tables in migrations, raise cost limits, and remove blockers like long transactions.

• Add Swap to a Small Django Server

  Add a swap file to a 1-2 GB Django server so pip installs, migrations, and traffic spikes do not trigger the OOM killer: sizing, fstab, swappiness, Docker and cloud caveats, and when swap is the wrong fix.

• Speed Up Django With Template Fragment Caching

  Use Django template fragment caching in production: the cache tag with vary-on keys, per-user and per-language fragments, cached template loaders, invalidation with make_template_fragment_key, and avoiding stale or leaked content.

• When Async Django Views Pay Off in Production

  Decide when async views improve a production Django app: concurrent external API calls, streaming, and long-polling benefit, while ORM-heavy views usually do not. Measure, avoid blocking calls, and run async code under ASGI correctly.

• Speed Up Django REST Framework APIs

  Make Django REST Framework APIs faster in production: eliminate N+1 queries with select_related and prefetch_related, slim serializers, paginate everything, cache expensive endpoints, use values() for read-heavy lists, and measure with profiling.

• Fast Pagination for Large Django Tables

  Paginate large Django tables without slow OFFSET and COUNT queries: keyset (cursor) pagination on an indexed column, DRF CursorPagination, avoiding exact counts, admin tips, and when page numbers are still fine.

• Fast Bulk Inserts and Updates in Django

  Speed up large Django data loads and updates: bulk_create and bulk_update with batch_size, update() with F expressions, update_conflicts for upserts, transactions, disabling per-row signals, COPY for very large imports, and what bulk methods skip.

• Tune PostgreSQL Settings for a Django Workload

  Tune PostgreSQL configuration for a Django app: shared_buffers, effective_cache_size, work_mem, max_connections with pooling, random_page_cost on SSDs, checkpoints and WAL, autovacuum, and statement timeouts, with safe starting values by server size.

• Serve Optimised Images and Thumbnails From Django

  Serve fast, right-sized images from a Django app: generate thumbnails in background tasks, use WebP and AVIF, responsive srcset, CDN or image service resizing, cache headers for media, and avoid resizing images inside web requests.

• Use Gunicorn preload_app With Django

  Decide whether to enable Gunicorn preload_app for Django: memory savings from copy-on-write, faster worker boot, fail-fast startup, the database and client connections you must close after fork, and why reloads need a full restart.

• Tune Nginx Keepalive and Worker Connections for Django

  Tune Nginx for a Django backend: worker_processes and worker_connections, client keepalive_timeout, upstream keepalive connection pools to Gunicorn or Uvicorn, proxy buffering, and file descriptor limits, with when each setting matters.

• Scale Django Channels WebSockets Across Servers

  Scale Django Channels beyond one process: the Redis channel layer and its capacity settings, running several Daphne or Uvicorn instances behind a load balancer, sticky sessions, connection limits, group fan-out costs, and graceful reconnects during deploys.

• Autoscale Django on Kubernetes With the HPA

  Autoscale Django web pods on Kubernetes with the Horizontal Pod Autoscaler: accurate CPU requests, scaling on CPU or requests per second, min and max replicas, scale-down stabilisation, database connection limits, and testing under load.

• Automatic ORM Query Caching in Django With Cachalot or Cacheops

  Cache Django ORM queries automatically with django-cachalot or django-cacheops: how invalidation works, when it speeds up read-heavy apps, which tables to exclude, multi-process and raw SQL pitfalls, and how to measure hit rates before trusting it in production.

• Prevent Cache Stampedes in Django

  Stop cache stampedes (dogpile effects) in Django when a popular cache key expires and hundreds of requests recompute it at once: use locks, stale-while-revalidate, jittered timeouts, background refresh, and probabilistic early expiration.

• A Practical Database Indexing Strategy for Django

  Decide which indexes a Django app needs: find slow queries, index foreign keys and filter and ordering columns, use composite, partial, covering, and expression indexes with Meta.indexes, remove unused indexes, and add them without locking production tables.

• Use QuerySet.explain() and EXPLAIN ANALYZE to Speed Up Django Queries

  Read PostgreSQL query plans for Django ORM queries with QuerySet.explain(analyze=True, buffers=True): spot sequential scans, bad row estimates, sorts, and nested loops, run EXPLAIN safely on production, and turn plans into indexes and queryset fixes.

• Index Django JSONField Queries With GIN on PostgreSQL

  Speed up Django JSONField lookups on PostgreSQL: choose GinIndex with jsonb_ops or jsonb_path_ops for containment queries, add expression B-tree indexes for specific keys, write queries that can use them, and build the indexes concurrently.

• Delete Old Rows in Django in Batches Without Hurting Production

  Implement data retention in Django: delete old logs, task results, sessions, and events in small batches with primary key ranges, avoid long locks and replica lag, handle cascades, schedule cleanup with Celery or cron, and consider partitioning for big tables.

• Auto Scale a Django Service on Amazon ECS

  Configure ECS service auto scaling for Django web and Celery services: target tracking on CPU or ALB requests per target, step scaling on SQS or custom queue metrics, sensible min and max tasks, cooldowns, database connection limits, and graceful shutdown.

• Set Kubernetes CPU and Memory Requests and Limits for Django

  Size Kubernetes resources for Django web and Celery pods: measure real usage, set memory requests near the working set and limits above peaks, avoid CPU throttling with sensible CPU limits, match Gunicorn workers to CPU requests, and use VPA recommendations.

• Manage PostgreSQL Connections for Django on Serverless Platforms

  Avoid connection exhaustion when Django runs on Lambda, Cloud Run, Vercel, or autoscaling containers: use pooled endpoints (RDS Proxy, Neon, Supabase pooler, PgBouncer), set CONN_MAX_AGE and server-side cursor options correctly, and cap concurrency per instance.

• Speed Up Django JSON Responses With orjson

  Use orjson to make Django and Django REST Framework JSON serialization faster: an orjson renderer and parser for DRF, an ORJSONResponse for plain views, handling of datetimes, Decimals, and UUIDs, and when JSON encoding is actually your bottleneck.

Check it

Review lists to run before launch or after a change.

• Django Performance Checklist Before Launch

  A pre-launch performance checklist for Django: query counts and indexes, caching, static and media delivery, Gunicorn and database sizing, background work, timeouts, and a load test against targets.

• Django Traffic Spike Readiness Checklist

  Prepare a Django app for a launch, sale, TV mention, or viral post: load test the critical paths, cache anonymous pages at the edge, protect the database, scale workers and queues, add rate limits and graceful degradation, and staff the event.

• Django Load Balancer Configuration Checklist

  A checklist for putting Django behind a load balancer: health checks that pass host validation, forwarded headers and SECURE_PROXY_SSL_HEADER, correct client IPs, idle and request timeouts, connection draining, sticky sessions avoided, and TLS policy.

Fix it

Errors and symptoms, each traced to a cause and a fix.

• Fix a Slow First Request After a Django Deploy

  Fix slow first requests after a Django deploy or restart: measure cold versus warm, find heavy imports with -X importtime, precompile bytecode, preload, warm workers, and avoid cold starts.

• Fix a Slow Django Admin on Large Tables

  Speed up Django admin changelists and forms on tables with millions of rows: avoid full counts, add list_select_related, use autocomplete and raw ID fields, index search and ordering, and limit filters.

• Find and Fix a Django Memory Leak in Production

  Find and fix a Django memory leak in production: tell real leaks from normal Python growth, recycle Gunicorn workers, profile with tracemalloc, and fix DEBUG query logging, unbounded caches, and large querysets.

• Fix High CPU Usage on a Django Server

  Fix high CPU usage on a Django server: find whether Gunicorn, PostgreSQL, Celery, or something else is burning CPU, profile hot endpoints with py-spy, fix N+1 queries and missing indexes, and block abusive traffic.

• Gunicorn Worker Timeout in Django: Root Causes and Fixes

  A Gunicorn worker timeout in Django production usually means a worker process stopped responding before Gunicorn’s timeout limit expired.

• Fix Django 504 Gateway Timeout Behind Nginx

  Diagnose Django 504 Gateway Timeout errors behind Nginx: find the slow layer, align proxy and Gunicorn timeouts, fix slow views, and offload long work.

• Fix PostgreSQL "Too Many Connections" in Django

  Fix PostgreSQL sorry, too many clients already errors from Django: count connections by source, fix CONN_MAX_AGE and worker math, find idle leaks, and add pooling.

• Fix Gunicorn Workers Killed by the OOM Killer

  Diagnose Gunicorn workers killed for running out of memory: OOM evidence, memory per worker, large querysets, leaks, max-requests, and right-sizing.

• Fix Load Balancer Health Checks Failing for Django

  Fix load balancer health checks that mark healthy Django servers down: DisallowedHost 400s, HTTPS redirects, auth on health paths, and slow checks.

• Fix Celery WorkerLostError in Django

  Fix Celery WorkerLostError 'Worker exited prematurely: signal 9 (SIGKILL)' in Django: find OOM kills and short stop timeouts, cap memory per child, chunk big tasks, and redeliver safely.

• Fix Nginx 499 Client Closed Request with Django

  Understand and fix Nginx 499 status codes in front of Django: find the slow endpoints and impatient clients, align load balancer and proxy timeouts, and move long work to background tasks.

• Fix Celery SoftTimeLimitExceeded and Hard Time Limits

  Fix Celery SoftTimeLimitExceeded and TimeLimitExceeded errors in Django: find why tasks run long, set soft and hard limits that fit the work, clean up on the soft limit, and split slow jobs into chunks.

• Fix a Celery Worker Whose Memory Keeps Growing

  Fix Celery worker memory that grows until the OOM killer strikes: recycle children with max-tasks-per-child and max-memory-per-child, find the task that leaks, and avoid DEBUG query logging and large result sets.

• Fix "Too many open files" on a Django Server

  Fix OSError Errno 24 Too many open files in Gunicorn, Celery, and Nginx: read the real per-process limit, find descriptor leaks with lsof, and raise LimitNOFILE, worker_rlimit_nofile, or Docker ulimits.

• Fix Django Cache Not Working in Production

  Fix a Django cache that never hits in production: confirm the configured backend, avoid LocMemCache across processes, understand why Vary Cookie and CSRF break per-view caching, and check keys, versions, and timeouts.

• Fix Cloudflare 524 Timeouts on Slow Django Requests

  Fix Cloudflare Error 524 a timeout occurred for slow Django views: understand the 100-second proxy limit, move exports and reports to Celery with polling, stream responses, and bypass the proxy for genuinely long admin tasks.

• Fix Heroku H12 Request Timeout in Django

  Fix Heroku H12 Request timeout errors for Django: understand the 30-second router limit, find slow views, set Gunicorn timeouts below it, move long work to worker dynos with Celery or RQ, and avoid H13 cascades.

• Fix "Canceling Statement Due to Conflict With Recovery" on Django Replicas

  Fix the PostgreSQL error canceling statement due to conflict with recovery when Django reads from a hot standby: understand vacuum conflicts, tune max_standby_streaming_delay and hot_standby_feedback, and route long reports to the right replica.

• Fix Heroku R14 "Memory Quota Exceeded" for a Django App

  Fix Heroku R14 Memory quota exceeded and R15 errors for Django: measure memory per Gunicorn worker, set WEB_CONCURRENCY to fit the dyno, recycle workers with max-requests, find leaks in views and Celery, and choose a dyno size that matches.

• Fix Nginx "worker_connections Are Not Enough" in Front of Django

  Fix the Nginx alert worker_connections are not enough while connecting to upstream for Django: count client and upstream connections, raise worker_connections and worker_rlimit_nofile, use upstream keepalive, and find slow responses that hold connections open.

Learning paths through this topic

• Docker, from image to productionBuild a production image, run it with Compose, ship it from CI, and check the result.
• Background jobs with CelerySet up workers and Beat, keep them supervised, route and tune them, and make tasks safe to retry.
• Django on KubernetesRun Django on a cluster with config, ingress, probes, jobs, graceful shutdown, and autoscaling, then package it with Helm and GitOps.
• Making Django fastMeasure first, then fix queries, caching, workers, and delivery, and prove it with a load test.

Related topics