Run Django-Q2 Workers and Schedules in Production

Problem statement

Django-Q2, the maintained fork of Django-Q, gives Django projects background tasks, schedules managed in the admin, and result storage in the database, without a separate scheduler process. It can use the Django ORM as its broker, which means no Redis to run. Production use still needs careful settings: the relationship between timeout and retry, worker counts, which broker to use at higher volume, and how to supervise qcluster.

1. pip install django-q2 and add django_q to INSTALLED_APPS, then migrate.
2. Configure Q_CLUSTER with a name, workers, timeout, and a retry value larger than timeout.
3. Use the ORM broker ("orm": "default") for low volume, or Redis for higher throughput.
4. Run python manage.py qcluster as one supervised process per server; manage schedules in the admin.
5. Keep save_limit and result cleanup in check so the success table does not grow without bound.

Step-by-step solution

1. Configure the cluster

INSTALLED_APPS += ["django_q"]

Q_CLUSTER = {
    "name": "shop",
    "workers": 4,
    "timeout": 120,
    "retry": 180,
    "max_attempts": 3,
    "queue_limit": 100,
    "save_limit": 500,
    "orm": "default",
    "catch_up": False,
}

timeout kills a task that runs too long; retry is how long the broker waits before redelivering an unacknowledged task. retry must be larger than timeout, or tasks still running get handed to another worker and run twice. catch_up = False stops the scheduler from running every missed occurrence after downtime.

2. Enqueue tasks and schedules

from django_q.tasks import async_task, schedule
from django_q.models import Schedule

async_task("orders.tasks.send_receipt", order.pk)

schedule("reports.tasks.nightly_summary", schedule_type=Schedule.CRON, cron="0 3 * * *")

Schedules live in the database and can be edited in the Django admin, which suits teams that want non-developers to adjust timings. Create them in a data migration or management command so environments match.

3. Choose the broker

• ORM broker: uses a table in your database. No extra service, transactional with your data, fine for hundreds of tasks per minute.
• Redis broker: "redis": os.environ["REDIS_URL"] instead of "orm". Better for high volume and lower latency.

With the ORM broker, polling queries add load; keep poll at its default or higher, and make sure the database can handle it.

4. Run qcluster

# /etc/systemd/system/shop-qcluster.service
[Service]
User=deploy
WorkingDirectory=/srv/app
EnvironmentFile=/etc/shop/env
ExecStart=/srv/app/.venv/bin/python manage.py qcluster
Restart=always
TimeoutStopSec=60

The cluster includes a sentinel, workers, a monitor, and the scheduler. Running clusters on several servers is supported; the scheduler uses database locking so schedules are not double-enqueued, but test this with your broker and version before relying on it.

5. Keep tables tidy

Successful tasks are stored up to save_limit; failed tasks are kept until you delete them. Review failures in the admin and clean up old records periodically. Set "ack_failures": True if failed tasks should not be retried endlessly by the broker.

Explanation

qcluster forks worker processes that pull tasks from the broker, run them with a timeout, and store results through the ORM. A monitor process saves results and a scheduler process checks the Schedule table each cycle and enqueues due tasks. Keeping everything in the Django database makes Django-Q2 easy to operate and inspect from the admin, at the cost of throughput compared with dedicated brokers.

Edge cases / notes

• Fork safety. Workers are forked processes; avoid database queries at import time so children do not inherit connections.
• Long tasks. Tasks longer than a few minutes need a high timeout and matching retry; consider splitting them.
• Original Django-Q. The original package is unmaintained; Django-Q2 is the maintained fork with support for current Django versions.

Next steps

• Compare options in Celery Alternatives for Django.
• Another database-backed option: Run Procrastinate as a Postgres Task Queue.
• Supervise processes with Run Celery With systemd for Django, which applies to qcluster too.

FAQ

Can Django-Q2 run without Redis?

Yes. The ORM broker stores tasks in your database.

Why do my tasks run twice?

retry is shorter than the task's runtime, so the broker redelivers it while it is still running. Make retry larger than timeout.

Where are schedules stored?

In the Schedule model, editable in the Django admin.

Is Django-Q2 the same as Django-Q?

It is a maintained fork of Django-Q with updates for current Django and Python versions.