Background jobs with Celery
Set up workers and Beat, keep them supervised, route and tune them, and make tasks safe to retry.
This path sets up Celery for Django and then makes it reliable: the broker, workers and Beat under systemd, separate queues, tuning, idempotent tasks with backoff retries and rate limits, broker persistence, and monitoring with Flower and Prometheus.
The early steps get tasks running; the later steps are what keep them running correctly under load and through deploys.
The steps
- Configure Redis and Celery for Django Production
Running background jobs in Django production is not just a matter of installing Celery and pointing it at Redis.
- Run Celery Workers with systemd for Django
Running Celery workers manually in production is fragile. If you start a worker from a shell and disconnect, the process may stop.
- Run Celery Beat for Scheduled Tasks in Django Production
Run Celery Beat reliably for Django: schedules, file or database schedulers, a single instance under systemd, time zones, and monitoring.
- Route Celery Tasks to Separate Queues
Stop slow Celery tasks from delaying urgent ones in Django: define queues, route tasks with CELERY_TASK_ROUTES, run dedicated workers per queue with their own concurrency and limits, and monitor queue depth.
- Tune Celery Worker Concurrency and Prefetch for Django
Tune Celery workers for Django workloads: pool types, concurrency, prefetch multiplier, acks_late, time and memory limits, and dedicated queues.
- Make Celery Tasks Idempotent and Safe to Retry
Design Django Celery tasks that can run twice without harm: enqueue on commit, idempotency keys and unique constraints, row locks, external API idempotency, and bounded retries with backoff.
- Retry Celery Tasks With Exponential Backoff in Django
Configure Celery retries for Django tasks: autoretry_for with retry_backoff and jitter, manual self.retry with countdown, max_retries and retry_backoff_max, which exceptions to retry, idempotency, and dead-letter handling for tasks that keep failing.
- Rate Limit Celery Tasks in Django
Control how fast Celery tasks run in a Django app: per-worker rate_limit on tasks, why it is not global, global limits with a Redis token bucket or dedicated queues and concurrency, handling third-party API 429 responses, and protecting the database from task floods.
- Configure Redis Persistence for Django and Celery
Decide which Redis data must survive restarts in a Django stack: AOF versus RDB for Celery queues and sessions, no persistence for caches, fork memory, Docker volumes, and verifying recovery.
- Monitor Celery with Flower
Run Flower to monitor Celery in a Django deployment: enable task events, run Flower under systemd bound to localhost, protect it with authentication behind Nginx, persist state, and export Prometheus metrics.
- Monitor Celery With Prometheus Metrics and Grafana
Expose Celery metrics to Prometheus for a Django app: run celery-exporter for task counts, failures, runtimes, and queue lengths, enable task events, build Grafana panels, and alert on growing queues, rising failures, and missing workers.
- Celery Production Checklist for Django
Review Celery before production: broker durability, queues and routing, retries and idempotency, time limits, single Beat, monitoring, deploys, and security.