Run django-rq Workers in Production

Problem statement

RQ (Redis Queue) is a simpler alternative to Celery: jobs are Python function calls stored in Redis, executed by worker processes, with less configuration and fewer concepts. django-rq integrates it with Django settings, the admin, and management commands. Many projects that only need "send this email later" or "process this upload in the background" find it easier to reason about. Production use still needs supervised workers, timeouts, failure handling, scheduling, and monitoring.

1. pip install django-rq, add django_rq to INSTALLED_APPS, and configure RQ_QUEUES with your Redis URL.
2. Enqueue with django_rq.enqueue(func, args) or the @job("default") decorator, inside transaction.on_commit when jobs read new rows.
3. Run workers with python manage.py rqworker default high under systemd (or as a container), restarting on failure.
4. Set default_timeout per queue and job_timeout per job; failed jobs go to the failed job registry.
5. Schedule recurring jobs with rq-scheduler (or RQ's built-in scheduler) or with systemd timers.
6. Watch queues in the Django admin and export metrics with rq-exporter if you use Prometheus.

Step-by-step solution

1. Configure queues

INSTALLED_APPS += ["django_rq"]

RQ_QUEUES = {
    "high": {"URL": env("REDIS_URL"), "DEFAULT_TIMEOUT": 60},
    "default": {"URL": env("REDIS_URL"), "DEFAULT_TIMEOUT": 300},
    "low": {"URL": env("REDIS_URL"), "DEFAULT_TIMEOUT": 1800},
}

# urls.py
path("django-rq/", include("django_rq.urls")),

The admin integration shows queues, workers, and failed jobs for staff users.

2. Enqueue jobs safely

import django_rq
from django.db import transaction

def signup(request):
    user = create_user(request.POST)
    transaction.on_commit(lambda: django_rq.enqueue(send_welcome_email, user.pk))

Pass IDs, not model instances; RQ pickles arguments. Use on_commit so the worker never runs before the row is committed.

3. Run workers under systemd

# /etc/systemd/system/rqworker@.service
[Unit]
Description=RQ worker %i
After=network.target redis-server.service

[Service]
User=deploy
WorkingDirectory=/srv/app/current
EnvironmentFile=/etc/myapp/env
ExecStart=/srv/app/current/.venv/bin/python manage.py rqworker high default low --with-scheduler
Restart=always
KillSignal=SIGTERM
TimeoutStopSec=600

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now rqworker@1 rqworker@2

Queue order matters: a worker listening on high default low always takes high jobs first. Each worker runs one job at a time in a forked work horse process, so run several worker instances for concurrency. After deploys, restart workers so they load new code.

4. Timeouts and failures

django_rq.enqueue(build_report, report_id, job_timeout=1200, result_ttl=3600, failure_ttl=7 * 86400)

A job exceeding its timeout is killed and moved to the FailedJobRegistry with its traceback. Retry transient failures automatically:

from rq import Retry
django_rq.enqueue(call_partner_api, order_id, retry=Retry(max=3, interval=[10, 60, 300]))

Review and requeue failed jobs from the admin or with rq requeue --all after fixing the cause.

5. Scheduling

Start workers with --with-scheduler (RQ 1.2+) to run jobs enqueued with enqueue_in or enqueue_at. For recurring cron-style jobs, use rq-scheduler with python manage.py rqscheduler, or keep recurring work in systemd timers that call management commands, which is simple and observable.

6. Monitoring

Watch queue length and failed job count. The admin page is enough for small setups; rq-exporter exposes Prometheus metrics for alerting on backlog and failures.

Verification check: python manage.py rqstats shows queues and workers, a test job enqueued from a Django shell completes within seconds, and a deliberately failing job appears in the failed registry with its traceback.

Explanation

RQ stores each job as a Redis hash with the pickled function reference and arguments, and pushes its ID onto a list per queue. A worker blocks on those lists, forks a work horse for each job, and records the outcome in registries (started, finished, failed, scheduled). Forking per job isolates memory leaks and crashes but limits each worker to one job at a time. django-rq adds Django-aware configuration, management commands, and admin views, so the whole setup stays within familiar Django tooling.

Edge cases / notes

• Windows is not supported by RQ workers because they fork; use WSL or Linux.
• Redis persistence. Jobs live in Redis; enable AOF or RDB so a restart does not lose queued jobs.
• Large arguments bloat Redis; store data in the database and pass IDs.
• Django tasks. Django's built-in tasks framework can use RQ-based backends; check compatibility with your Django version.

Next steps

• The Celery equivalent: Configure Redis and Celery for Django in Production.
• Django's own interface: Deploy Django Tasks With Background Workers.

FAQ

Is django-rq good enough for production?

Yes, for typical background jobs, when workers are supervised, timeouts are set, and failures are monitored.

How do I run several RQ workers?

Start several worker processes, for example with a templated systemd unit like rqworker@1 and rqworker@2.

How do I schedule recurring jobs with django-rq?

Use rq-scheduler or systemd timers calling management commands.

What happens when a job times out?

The work horse is killed and the job moves to the failed registry with its traceback.