Fix Celery SoftTimeLimitExceeded and Hard Time Limits

Problem statement

A Celery task that used to finish now fails with one of these:

celery.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()

[ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(300,)
Hard time limit (300s) exceeded for reports.tasks.build_export[...]

The soft limit raises an exception inside the task so it can clean up. The hard limit kills the child process outright, which also shows up as WorkerLostError and a restarted pool process. Both mean the same thing: the task ran longer than you allowed. The fix is either making the task shorter, giving it a limit that matches the real work, or both, not raising the limit until the error goes away.

1. Find the limits in effect: task_soft_time_limit, task_time_limit, and any per-task soft_time_limit=.
2. Measure how long the task really takes using worker logs or Flower runtimes.
3. Set a soft limit above the normal 99th percentile and a hard limit 30-60 seconds above the soft one.
4. Catch SoftTimeLimitExceeded to save progress or release locks, then re-raise or retry.
5. Split long jobs into smaller tasks so no single task needs a long limit.
6. Remember time limits do not work with the solo or threads pools in the same way as prefork.

Step-by-step solution

1. Find which limit fired and where it is set

Limits come from global settings or per-task options:

# settings.py
CELERY_TASK_SOFT_TIME_LIMIT = 240
CELERY_TASK_TIME_LIMIT = 300

@shared_task(soft_time_limit=600, time_limit=660)
def build_export(export_id):
    ...

Per-task values override globals, and a caller can override both with apply_async(soft_time_limit=...). Search the codebase for all three before changing anything.

2. Measure the real runtime

Worker logs at info level record each task's runtime:

Task reports.tasks.build_export[...] succeeded in 212.4s

Collect these for a day or check task runtimes in Flower. You want the typical and worst normal runtime. A task that normally takes 20 seconds and occasionally takes 400 has a different problem (a slow query, an external API hanging) from one that steadily grows as data grows.

3. Add timeouts inside the task

Many soft-limit errors are a hung network call. Set explicit timeouts on everything the task waits for:

import requests

resp = requests.get(url, timeout=(5, 30))   # connect, read

For database work, a PostgreSQL statement_timeout on the worker's connection stops one query from consuming the entire limit. The task then fails with a clear error instead of an opaque time limit.

4. Clean up on the soft limit

The soft limit exists so the task can leave things in a good state:

from celery.exceptions import SoftTimeLimitExceeded

@shared_task(bind=True, soft_time_limit=240, time_limit=300, max_retries=3)
def build_export(self, export_id):
    export = Export.objects.get(pk=export_id)
    try:
        for chunk in export.pending_chunks():
            write_chunk(export, chunk)
            export.mark_chunk_done(chunk)
    except SoftTimeLimitExceeded:
        export.status = "partial"
        export.save(update_fields=["status"])
        raise self.retry(countdown=10)

Because progress is saved per chunk, the retry resumes instead of restarting. This only works if the task is idempotent; see Make Celery Tasks Idempotent and Safe to Retry.

5. Split long jobs into chunks

A task that processes 200,000 rows should not be one task. Fan out:

from celery import group

@shared_task
def export_rows(export_id, start, stop):
    ...

def start_export(export_id, total, size=5000):
    group(export_rows.s(export_id, i, i + size) for i in range(0, total, size)).apply_async()

Each child task finishes in seconds, retries independently, and spreads across workers. Use a chord if something must run after all chunks complete.

6. Route long tasks away from short ones

Even with correct limits, a 10-minute task occupies a worker slot. Put slow tasks on their own queue with their own workers so quick tasks such as emails are not stuck behind them. See Route Celery Tasks to Separate Queues.

Verification check: after deploying, watch runtimes in the worker log or Flower for a full business cycle. No SoftTimeLimitExceeded should appear for normal inputs, and any that do should leave the record in a resumable state.

Explanation

In the prefork pool, the soft limit is implemented with a signal that raises SoftTimeLimitExceeded inside the running task. The hard limit is enforced by the parent process, which terminates the child and replaces it. Because the hard limit is a kill, nothing in the task runs afterwards: database transactions roll back, files may be half written, and locks taken in Redis stay until they expire. That is why the soft limit and a cleanup handler matter more than the exact numbers.

Edge cases / notes

• Pools. Time limits rely on signals and child processes. With --pool=threads, gevent, or eventlet, the hard limit is not enforced the same way, and the soft limit may not be either. Use I/O timeouts in those pools.
• Windows. Signals used by the soft limit are not available on Windows.
• Visibility timeout. With Redis as broker, a task that runs longer than visibility_timeout can be delivered twice. Keep the hard limit well below it.
• Catching too broadly. A bare except Exception also catches SoftTimeLimitExceeded, so the task carries on until the hard kill. Catch it explicitly first.

Next steps

• Size worker concurrency for long tasks with Tune Celery Workers for Django.
• If children are killed for memory rather than time, see Fix Celery WorkerLostError.

FAQ

What is the difference between soft and hard time limits?

The soft limit raises an exception inside the task so it can clean up. The hard limit kills the worker child process with no chance to run more code.

Should the hard limit always be higher than the soft limit?

Yes. If they are equal or the soft limit is missing, the task is killed without warning.

Why does a hard limit cause WorkerLostError?

The parent kills the child process running the task, and Celery reports the lost child as WorkerLostError.

Can I disable time limits?

You can, but a single hung call then blocks a worker slot forever. Keep limits and add timeouts instead.