Fix Celery Tasks Running Old Code After a Deploy

Problem statement

You deployed a fix to a Celery task, the web app shows the new code, but tasks still behave the old way: the bug persists, new log lines never appear, or tasks fail with TypeError: send_invoice() got an unexpected keyword argument 'locale'. Some tasks run the new code and some the old.

Celery workers import task modules when they start and keep them in memory. Deploying new files changes nothing for a running worker. Unless every worker process on every host is restarted as part of the deploy, and unless messages queued by the old code are compatible with the new code (and the other way round), you get old behaviour or signature errors. This guide finds stale workers and makes deploys update them reliably.

1. List workers and their uptime: celery -A config inspect stats and check each worker's start time against the deploy time.
2. Check which directory each worker runs from: ls -l /proc/<pid>/cwd; release-directory deploys need a restart, not a reload.
3. Restart every worker (and Beat) in the deploy, gracefully: systemctl restart celery-worker with a stop timeout longer than your tasks.
4. Include all hosts and all worker types (separate queues, separate services) in the restart step.
5. Keep task signatures backward compatible for one release: add parameters with defaults, never remove or rename in the same deploy.
6. For breaking changes, create a new task name and let the old queue drain.

Step-by-step solution

1. See which workers are running and since when

celery -A config inspect stats --json | python3 -c "
import json, sys, time
for name, s in json.load(sys.stdin).items():
    pid = s.get('pid'); up = s.get('uptime')
    print(f'{name}: pid={pid} uptime={up}s started~{time.strftime(\"%H:%M:%S\", time.localtime(time.time()-up))}')
"

Any worker started before your deploy is running old code. celery inspect registered lists the task names each worker knows, which shows missing new tasks.

Log the release version in tasks to make this obvious in logs:

# config/celery.py
import os

from celery.signals import worker_ready

RELEASE = os.environ.get("RELEASE", "unknown")


@worker_ready.connect
def announce(sender, **kwargs):
    sender.app.log.get_default_logger().info("worker ready, release=%s", RELEASE)

2. Check what code each worker loaded

For release-directory deploys (/srv/myapp/current -> releases/<timestamp>), a process resolves the symlink when it starts. Check each worker:

for pid in $(pgrep -f "celery.*worker"); do echo "$pid $(readlink /proc/$pid/cwd)"; done
readlink -f /srv/myapp/current

Workers pointing at an old release directory need a restart. They keep working even after that directory is deleted, still running the deleted code.

3. Restart workers in every deploy

sudo systemctl restart celery-worker celery-beat

A restart sends SIGTERM, Celery performs a warm shutdown (finishes running tasks, stops taking new ones), and systemd starts a fresh worker with the new code. Configure the unit so the warm shutdown has time to complete:

[Service]
KillSignal=SIGTERM
TimeoutStopSec=600

Details that commonly go wrong:

• Multiple services: workers for separate queues (celery-worker-emails, celery-worker-reports) each need restarting. Use a systemd target or template units so one command covers them all.
• Multiple hosts: the deploy must restart workers on every host that runs them, not only on the web servers.
• Containers: redeploying the web service does not update a separate worker service or Deployment. Roll out the worker image with the same tag in the same release.
• celery multi: celery multi restart must be given the same node names and options as start.

Avoid relying on celery control pool_restart; it is disabled by default and does not reliably reload changed modules.

4. Keep old and new code compatible

During a deploy, the web app (new code) can enqueue tasks that an old worker executes, and tasks enqueued before the deploy can be executed by a new worker. Messages carry the task name and arguments, not code. Rules that keep this safe:

• Add parameters with defaults, so old messages without them still work:

@app.task
def send_invoice(invoice_id, locale="en"):
    ...

• Do not remove or rename parameters in the same release that stops sending them. First deploy code that accepts both, then remove the old one later.
• Do not rename tasks in place. The task name (by default the module path plus function name) is how workers find the code; moving a function to another module changes its name. Set explicit names (@app.task(name="billing.send_invoice")) for important tasks so refactors do not break queued messages.
• Breaking changes: add a new task (send_invoice_v2), switch callers to it, and remove the old one after its queue is empty.

5. Deal with ETA and countdown tasks

Tasks scheduled for the future (apply_async(countdown=3600)) are held by workers in memory or sit in the broker until due, then run with whatever code the executing worker has. They must stay compatible for as long as they can be delayed.

Verification check:

celery -A config inspect stats --json | python3 -c "import json,sys; [print(k, v['uptime']) for k, v in json.load(sys.stdin).items()]"
journalctl -u celery-worker --since "10 minutes ago" | grep "release="

Every worker's uptime should be shorter than the time since the deploy, and the log should show the new release version.

Explanation

Python imports a module once per process; Celery workers import task modules at start-up and on their first use, and then execute the in-memory functions. The prefork pool's child processes inherit the parent's imported modules, so even when children are recycled they bring back the same old code. New code arrives only through a new worker process started after the files changed. Because the broker decouples producers and consumers, versions mix during every deploy, which is why compatible task signatures matter as much as restarts.

Edge cases / notes

Celery Beat

Beat reads the schedule at start-up (and, with django-celery-beat, from the database). Code changes to periodic task definitions need a Beat restart too. Run exactly one Beat instance.

Long-running tasks and deploys

A warm shutdown waits for running tasks. If tasks can run for an hour, deploys either wait an hour for workers to stop or kill tasks. Split long work into smaller tasks, or give long-running queues their own workers with a separate restart policy.

When to automate this

Put the worker restart in the same deploy script as the web restart, driven by a list of all worker services and hosts. After restarting, compare worker uptimes against the deploy time and fail the deploy if any worker is older.

Next steps

• Supervise workers properly with Run Celery Workers with systemd for Django.
• For the web side of the same problem, see Fix Django Code Changes Not Showing After Deploy.
• Make tasks safe when they run twice or late with Make Celery Tasks Idempotent and Safe to Retry.
• Review worker operations with the Celery Production Checklist for Django.

FAQ

Why don't Celery workers pick up new code after a deploy?

Workers keep the code they imported at start-up. They must be restarted after new code is deployed.

Is there a way to reload Celery workers without restarting?

Not reliably. Restart workers with a warm shutdown so running tasks finish first.

Why do Celery tasks fail with "unexpected keyword argument" after a deploy?

An old worker received a message from new code with a new parameter, or the other way round. Add parameters with defaults and restart all workers.

Does renaming a Celery task break queued messages?

Yes. Messages reference tasks by name. Keep explicit task names stable and introduce new names for breaking changes.