Run Procrastinate as a PostgreSQL Task Queue for Django

Problem statement

You already run PostgreSQL and would rather not add Redis or RabbitMQ just for background jobs. Procrastinate is a task queue that stores jobs in PostgreSQL and uses LISTEN/NOTIFY and SELECT ... FOR UPDATE SKIP LOCKED to distribute them to workers. Because jobs live in the same database as your data, deferring a job inside a transaction is atomic: if the transaction rolls back, the job disappears too. Production use needs a worker process, sensible concurrency, periodic tasks, and attention to database load.

1. pip install "procrastinate[django]", add procrastinate.contrib.django to INSTALLED_APPS, and run migrate.
2. Define tasks with @app.task from procrastinate.contrib.django and defer them with task.defer(...).
3. Run workers with python manage.py procrastinate worker --concurrency 4.
4. Declare periodic tasks with @app.periodic(cron="0 3 * * *"); workers schedule them without a separate beat process.
5. Use lock and queueing_lock to prevent overlapping or duplicate jobs, and clean old jobs regularly.

Step-by-step solution

1. Install and migrate

INSTALLED_APPS += ["procrastinate.contrib.django"]

python manage.py migrate

Procrastinate's tables, functions, and triggers are created through Django migrations, so they deploy with the rest of your schema. It uses the default database connection settings.

2. Define and defer tasks

# orders/tasks.py
from procrastinate.contrib.django import app

@app.task(queue="emails", retry=3)
def send_receipt(order_id: int):
    order = Order.objects.get(pk=order_id)
    ...

from django.db import transaction

with transaction.atomic():
    order = Order.objects.create(...)
    send_receipt.defer(order_id=order.pk)

Because the job row is inserted in the same transaction as the order, a rollback removes both. There is no race where a worker picks up a job for an order that is not committed, which is the classic problem transaction.on_commit() works around with Celery.

3. Run workers

python manage.py procrastinate worker --concurrency 4 --queues emails,default

Workers use asynchronous I/O internally; synchronous Django tasks run in threads, so --concurrency sets how many run at once. Each concurrent synchronous task may hold a database connection. Supervise workers with systemd or as container services, and give them a stop timeout so running jobs finish on SIGTERM.

4. Periodic tasks

@app.periodic(cron="0 3 * * *")
@app.task(queueing_lock="nightly-report")
def nightly_report(timestamp: int):
    ...

Workers coordinate periodic scheduling through the database, so several workers do not create duplicate runs for the same timestamp. The task receives the scheduled timestamp as an argument.

5. Locks, retention, and monitoring

• lock="customer-42" ensures jobs with the same lock run one at a time.
• queueing_lock="sync-customer-42" rejects a new job while one with the same key is still waiting.
• Jobs remain in the table after completion; delete old finished jobs periodically (Procrastinate provides commands or delete_old_jobs for this) to keep the table small.
• The Django admin integration shows jobs and their states.

Explanation

Procrastinate inserts each job as a row and notifies listening workers through PostgreSQL's NOTIFY. Workers claim jobs with SELECT ... FOR UPDATE SKIP LOCKED, which lets many workers take different rows without blocking each other, and update the row when the job finishes or fails. All coordination, including locks and periodic deduplication, happens in SQL, so the only infrastructure is the database you already have.

Edge cases / notes

• PgBouncer. LISTEN/NOTIFY does not work through transaction-pooling PgBouncer; give workers a direct connection or session pooling.
• Database load. Every job is several writes; at very high volume this competes with application queries. Monitor table bloat and autovacuum on the jobs table.
• Async tasks. Procrastinate also supports async def tasks for I/O-heavy work.

Next steps

• Compare queues in Celery Alternatives for Django.
• Another database-backed option: Run Django-Q2 Workers in Production.
• Achieve the same consistency with Celery through Transactional Outbox With Django and Celery.

FAQ

Does Procrastinate need Redis?

No. It uses PostgreSQL for storing jobs and notifying workers.

Is deferring a job inside a transaction safe?

Yes. The job is a row in the same transaction, so it commits or rolls back with your data.

Can I run several workers?

Yes. They share the jobs table and use row locking to avoid taking the same job.

Does it work with PgBouncer?

Workers need a connection that supports LISTEN/NOTIFY, so use direct or session-pooled connections for them.