Monitor Celery With Prometheus Metrics and Grafana

Problem statement

Celery problems are silent: a queue grows for hours because workers died, a task starts failing after a deploy, or runtimes creep up until timeouts hit. Flower shows the current state but is not built for history or alerting. Prometheus with a Celery exporter records task rates, failures, runtimes, and queue lengths over time, Grafana visualizes them, and Alertmanager tells you when something drifts.

1. Enable task events: worker_send_task_events = True and task_send_sent_event = True.
2. Run celery-exporter (for example danihodovic/celery-exporter) pointed at the broker.
3. Scrape its /metrics endpoint with Prometheus.
4. Graph task rates by state, runtime percentiles, queue length, and worker count in Grafana.
5. Alert on queue length growth, failure ratio, and zero workers online.

Step-by-step solution

1. Enable events

CELERY_WORKER_SEND_TASK_EVENTS = True
CELERY_TASK_SEND_SENT_EVENT = True

Restart workers. Events are published to the broker for every task state change; the exporter consumes them. On very high task volumes, events add broker traffic; measure the overhead.

2. Run the exporter

services:
  celery-exporter:
    image: danihodovic/celery-exporter
    environment:
      CE_BROKER_URL: ${REDIS_URL}
    ports: ["127.0.0.1:9808:9808"]
    restart: unless-stopped

The exporter exposes metrics such as celery_task_sent_total, celery_task_succeeded_total, celery_task_failed_total, celery_task_runtime (histogram), celery_queue_length, and celery_worker_up, labeled by task name, queue, and hostname. Check the exporter's documentation for the exact names in your version.

3. Scrape with Prometheus

scrape_configs:
  - job_name: celery
    static_configs:
      - targets: ["celery-exporter:9808"]

In Kubernetes, add a ServiceMonitor (Prometheus Operator) or pod annotations depending on your setup.

4. Build Grafana panels

Useful queries:

sum by (name) (rate(celery_task_succeeded_total[5m]))
sum by (name) (rate(celery_task_failed_total[5m])) / sum by (name) (rate(celery_task_received_total[5m]))
histogram_quantile(0.95, sum by (le, name) (rate(celery_task_runtime_bucket[5m])))
max by (queue_name) (celery_queue_length)

Community dashboards for celery-exporter give a starting point; add deploy annotations so changes in failure rates line up with releases.

5. Alert

groups:
  - name: celery
    rules:
      - alert: CeleryQueueGrowing
        expr: max by (queue_name) (celery_queue_length) > 1000 and deriv(celery_queue_length[15m]) > 0
        for: 15m
      - alert: CeleryHighFailureRate
        expr: sum(rate(celery_task_failed_total[10m])) / sum(rate(celery_task_received_total[10m])) > 0.05
        for: 10m
      - alert: CeleryNoWorkers
        expr: sum(celery_worker_up) == 0
        for: 5m

Tune thresholds to your normal traffic; a nightly batch may legitimately build a queue.

Explanation

Celery workers emit events (task-sent, task-received, task-started, task-succeeded, task-failed, worker-heartbeat) over the broker when events are enabled. An exporter listens to those events, aggregates them into counters and histograms, and inspects queue lengths directly in the broker. Prometheus scrapes these on an interval, giving time series that show trends and support alert rules. This complements error tracking: Sentry tells you which exception happened; metrics tell you how often and whether it is getting worse.

Edge cases / notes

• Queue length with Redis. Queue length is the Redis list length; reserved (prefetched) tasks are not counted, so set a low prefetch multiplier for accurate backlog numbers.
• Cardinality. Task names as labels are fine; never add task arguments or IDs as labels.
• Beat. Monitor beat separately with a heartbeat task, since a stopped beat produces no failures, just silence.

Next steps

• Debug live with Monitor Celery With Flower.
• Cover Django itself with Monitor Django With Prometheus and Grafana.
• Act on what you see with Tune Celery Workers for Django.

FAQ

Does Celery have built-in Prometheus metrics?

No. Use an exporter such as celery-exporter, which consumes Celery events.

Do I need to enable task events?

Yes, for task-level metrics. Queue length can be read from the broker without events.

What should I alert on first?

Queue length growing, failure ratio above a threshold, and no workers online.

Is Flower still useful?

Yes, for live inspection and task actions. Prometheus is better for history and alerts.