Rate Limit Celery Tasks in Django

Problem statement

A newsletter send enqueues 50,000 email tasks; an import enqueues a task per row that calls a partner API limited to 10 requests per second. Workers run as fast as they can, the email provider throttles you, the partner returns HTTP 429, and the database sees a burst of writes. You need tasks to run at a controlled pace. Celery's built-in rate_limit helps, but it applies per worker, which surprises many teams when they scale out.

1. For a simple cap per worker, set @shared_task(rate_limit="10/s").
2. Remember it is per worker process node, not global: four workers each allow 10/s, so 40/s total.
3. For a global limit, route the tasks to a dedicated queue served by one worker with low concurrency, or use a Redis-based token bucket inside the task.
4. When an API returns 429, retry with backoff using the Retry-After header.
5. Spread large batches over time with countdown or ETA, or chunk them into fewer, bigger tasks.

Step-by-step solution

1. Use rate_limit for per-worker caps

@shared_task(rate_limit="10/s")
def sync_contact(contact_id):
    ...

Formats are N/s, N/m, or N/h. Each worker instance (each celery worker process, regardless of its concurrency) enforces the rate for that task type. You can change it at runtime: app.control.rate_limit("crm.tasks.sync_contact", "5/s"). With worker_disable_rate_limits = True, limits are ignored entirely.

2. Get a global limit with queue design

The simplest global limit is structural: put the rate-limited tasks on their own queue and give that queue exactly one worker with a known concurrency and a per-worker rate_limit.

CELERY_TASK_ROUTES = {"crm.tasks.sync_contact": {"queue": "crm"}}

celery -A config worker -Q crm --concurrency 2 -n crm@%h

Scaling out the default queue no longer affects the CRM rate. Keep exactly one replica of the crm worker.

3. Use a shared token bucket for strict limits

When several worker pools must share one API limit, coordinate through Redis:

import time
from django_redis import get_redis_connection

def acquire(key, rate_per_sec, burst):
    r = get_redis_connection("default")
    script = """
    local tokens = tonumber(redis.call('get', KEYS[1]) or ARGV[2])
    local last = tonumber(redis.call('get', KEYS[2]) or ARGV[3])
    local now = tonumber(ARGV[3])
    tokens = math.min(tonumber(ARGV[2]), tokens + (now - last) * tonumber(ARGV[1]))
    if tokens < 1 then return 0 end
    redis.call('set', KEYS[1], tokens - 1, 'EX', 60)
    redis.call('set', KEYS[2], now, 'EX', 60)
    return 1
    """
    return r.eval(script, 2, f"{key}:tokens", f"{key}:ts", rate_per_sec, burst, time.time())

@shared_task(bind=True, max_retries=None)
def call_partner(self, item_id):
    if not acquire("partner-api", rate_per_sec=10, burst=10):
        raise self.retry(countdown=0.5)
    ...

The Lua script runs atomically in Redis, so all workers share one bucket. Libraries such as limits or pyrate-limiter with a Redis backend provide tested implementations.

4. Respect 429 responses

Even with limits, providers throttle. Retry with their guidance:

@shared_task(bind=True, max_retries=8)
def send_to_partner(self, payload):
    resp = requests.post(URL, json=payload, timeout=10)
    if resp.status_code == 429:
        raise self.retry(countdown=int(resp.headers.get("Retry-After", 30)))
    resp.raise_for_status()

5. Shape bursts at enqueue time

For batch jobs, spread tasks out instead of enqueuing everything at once, or group work:

for i, chunk in enumerate(chunked(contact_ids, 100)):
    sync_contacts_batch.apply_async(args=[chunk], countdown=i * 10)

Long countdowns keep messages in worker memory (with Redis and SQS visibility timeouts in play), so prefer a periodic task that pulls the next batch from the database for very long schedules.

Explanation

Celery's rate_limit is implemented inside each worker as a token bucket per task type: the worker delays executing tasks that would exceed the rate. Because workers do not share state, the effective global rate is the per-worker rate multiplied by the number of workers. Global limits require either a single consumer (queue design) or shared state (Redis). Retries with backoff handle the remaining throttling from external systems.

Edge cases / notes

• Prefetching. Rate-limited tasks are reserved by the worker while waiting; set worker_prefetch_multiplier = 1 to avoid one worker hoarding them.
• Autoscaling. Scaling workers up raises per-worker-limited throughput; keep rate-limited tasks on fixed-size pools.
• Priority. Rate-limited queues can back up; keep unrelated tasks on other queues so they are not delayed.

Next steps

• Handle failures with Celery Retry With Exponential Backoff.
• Design queues with Route Celery Tasks to Separate Queues.
• Size pools with Tune Celery Workers for Django.

FAQ

Is Celery's rate_limit global?

No. It applies per worker instance. Multiply by the number of workers to get the total.

How do I enforce a global rate across workers?

Use a dedicated queue with one worker, or a shared token bucket in Redis.

What should a task do when an API returns 429?

Retry after the Retry-After duration, with a cap on attempts.

Can I change a rate limit without redeploying?

Yes, with app.control.rate_limit(...), which updates running workers until they restart.