Run Huey as a Lightweight Django Task Queue

Problem statement

You need background tasks and a few scheduled jobs for a small or medium Django app, and Celery's configuration surface feels like too much. Huey is a small task queue with a built-in periodic scheduler, Django integration, and storage backends for Redis or even SQLite. In production, the main concerns are running the consumer reliably, choosing worker type and count, avoiding duplicate periodic tasks when you run more than one consumer, and keeping database connections healthy.

1. Install huey and redis, and add huey.contrib.djhuey to INSTALLED_APPS.
2. Configure HUEY in settings with the Redis connection, immediate = False in production, and consumer options.
3. Use @db_task and @db_periodic_task decorators, which close database connections around each task.
4. Run python manage.py run_huey under systemd or as a container; it starts workers and the scheduler.
5. If you run several consumers, disable the scheduler (periodic = False) on all but one.

Step-by-step solution

1. Configure Huey

INSTALLED_APPS += ["huey.contrib.djhuey"]

HUEY = {
    "huey_class": "huey.RedisHuey",
    "name": "shop",
    "immediate": False,
    "connection": {"url": os.environ["REDIS_URL"]},
    "consumer": {
        "workers": 4,
        "worker_type": "thread",
        "periodic": True,
        "health_check_interval": 1,
    },
}

immediate defaults to the value of DEBUG, which runs tasks inline during development. Set it explicitly to False in production settings so tasks go through Redis. For tiny deployments, huey.SqliteHuey with a file path works without Redis, but only for a single server.

2. Define tasks

from huey import crontab
from huey.contrib.djhuey import db_task, db_periodic_task

@db_task(retries=3, retry_delay=30)
def send_receipt(order_id):
    order = Order.objects.get(pk=order_id)
    ...

@db_periodic_task(crontab(minute="0", hour="3"))
def purge_expired_carts():
    Cart.objects.filter(expires_at__lt=timezone.now()).delete()

The db_ variants call close_old_connections() before and after each task, which prevents stale connection errors in long-running consumers.

3. Run the consumer

python manage.py run_huey --workers 4 --worker-type thread

Under systemd:

[Service]
User=deploy
WorkingDirectory=/srv/app
EnvironmentFile=/etc/shop/env
ExecStart=/srv/app/.venv/bin/python manage.py run_huey
Restart=always
KillSignal=SIGINT
TimeoutStopSec=60

Huey treats SIGINT as a graceful shutdown (finish running tasks) and SIGTERM as immediate; send SIGINT for clean deploys, which is why the unit sets KillSignal=SIGINT.

4. Scale without duplicate schedules

The scheduler runs inside the consumer. Two consumers with periodic = True enqueue every periodic task twice. When scaling out, run one consumer with the scheduler and others with --no-periodic (or "periodic": False):

python manage.py run_huey --no-periodic --workers 8

5. Monitor

Huey has no built-in dashboard. Log task start and failure through signals (@HUEY.signal(SIGNAL_ERROR)), export counts to your metrics system, and alert when the pending queue length in Redis grows.

Explanation

Huey stores tasks in Redis (or SQLite) and a consumer process pulls them off with a pool of threads, processes, or greenlets. The same process runs a scheduler that checks periodic tasks every minute and enqueues the ones that are due. That combined design is what makes Huey simple: one command gives you workers and cron-like scheduling. It also means scaling to several consumers requires turning the scheduler off on the extras.

Edge cases / notes

• Result storage. Huey stores task results in Redis by default; disable with "results": False if you never read them.
• Task locking. @HUEY.lock_task("name") prevents overlapping runs of long periodic tasks.
• SQLite storage. Works for one server with low volume; do not share a SQLite queue between hosts.

Next steps

• Compare options in Celery Alternatives for Django.
• Need more features? See Run Dramatiq Workers for Django.
• Consider Django's built-in tasks framework in Deploy Django Tasks Background Workers.

FAQ

Does Huey need Redis?

No. It also supports SQLite and in-memory storage, but Redis is the usual production choice.

How do I run periodic tasks with Huey?

Decorate them with @db_periodic_task and run the consumer with the scheduler enabled.

Why do my periodic tasks run twice?

Two consumers have the scheduler enabled. Start extra consumers with --no-periodic.

Why do tasks run immediately without the consumer?

immediate is true, which is the default when DEBUG = True. Set it to False in production.