Fix Celery Beat "Pidfile Already Exists" in Docker and systemd

Problem statement

After a container restart, a crash, or a server reboot, Celery beat refuses to start:

ERROR: Pidfile (celerybeat.pid) already exists.
Seems we're already running? (pid: 1)

Beat writes its process ID to celerybeat.pid when it starts and removes it on clean shutdown. If the process is killed (container stop timeout, OOM kill, power loss), the file stays. On the next start, beat sees the file, assumes another beat is running, and exits. In Docker this happens constantly because the PID inside a container is often 1, which always looks "alive".

1. In Docker, disable the pidfile: celery -A config beat --pidfile= (empty value).
2. Or point it at a path that is not persisted: --pidfile=/tmp/celerybeat.pid inside the container.
3. Under systemd, use RuntimeDirectory=celery and --pidfile=/run/celery/beat.pid; /run is cleared on boot.
4. Keep the schedule file separate: --schedule=/var/lib/celery/celerybeat-schedule in a persistent directory.
5. Make sure only one beat runs; the pidfile was never a reliable lock across machines.

Step-by-step solution

1. Find where the pidfile is written

Without --pidfile, beat writes celerybeat.pid in the current working directory. In Docker that is often /app, which may be a bind mount or a volume that survives restarts, so the stale file comes back every time.

docker compose exec beat ls -la /app | grep celerybeat

2. Disable it in containers

Docker or Kubernetes already guarantee one process per container and restart it if it dies. The pidfile adds nothing:

services:
  beat:
    image: myapp:latest
    command: celery -A config beat -l info --pidfile= --schedule=/tmp/celerybeat-schedule

An empty --pidfile= turns the pidfile off. With django-celery-beat, the schedule lives in the database, so use --scheduler django_celery_beat.schedulers:DatabaseScheduler and skip the schedule file entirely.

3. Use /run under systemd

# /etc/systemd/system/celery-beat.service
[Service]
User=deploy
Group=deploy
WorkingDirectory=/srv/app
RuntimeDirectory=celery
StateDirectory=celery
ExecStart=/srv/app/.venv/bin/celery -A config beat -l info \
  --pidfile=/run/celery/beat.pid \
  --schedule=/var/lib/celery/celerybeat-schedule
Restart=always

RuntimeDirectory creates /run/celery on start and removes it on stop, so a stale pidfile cannot survive. StateDirectory creates /var/lib/celery for the schedule file, which should persist so beat remembers when tasks last ran.

4. Separate the pidfile from the schedule file

The schedule file (celerybeat-schedule, sometimes with .db or .dat extensions) records last run times. It is fine to keep it across restarts. Deleting it occasionally causes some periodic tasks to run immediately on the next start. The pidfile, by contrast, is only meaningful while the process runs.

5. Guarantee a single beat

The original purpose of the pidfile was to stop two beats on one machine. It never protected against two beats on two machines, which causes duplicate periodic tasks. Run beat as exactly one replica (Kubernetes replicas: 1 with strategy: Recreate, one Compose service, one systemd unit), or use a scheduler with locking. See Fix Celery Beat Duplicate Tasks.

Explanation

Celery's beat uses a classic Unix pidfile: write your PID on start, check on startup whether a file exists and the PID is alive, remove it on exit. Inside containers, PIDs are namespaced and the main process is usually PID 1, so a stale file containing 1 points at the new beat itself, and the "is it alive?" check passes. Beat then concludes another instance is running and stops. Supervisors such as Docker, Kubernetes, and systemd already know whether the process runs, so the pidfile is unnecessary there.

Edge cases / notes

• Worker pidfiles. Celery workers started with celery multi use pidfiles too, with the same stale-file problem; prefer one systemd unit or container per worker.
• Read-only filesystems. Beat cannot write a pidfile into a read-only image directory; disabling it avoids that error as well.
• Graceful stops. Give beat time to exit (stop_grace_period in Compose) so it removes its own files.

Next steps

• Set up beat properly with Set Up Celery Beat for Django in Production.
• If beat starts but tasks never run, see Fix Celery Beat Not Sending Tasks.
• Write the unit files with Run Celery With systemd for Django.

FAQ

Is it safe to delete celerybeat.pid?

Yes, if no beat process is running. Better, configure beat so the file is not persisted at all.

Why does it always say pid 1?

Inside a container the main process is PID 1. The stale file contains 1, which the new process also has, so the alive check passes.

Should I delete the schedule file too?

No. It records last run times. Deleting it can make periodic tasks run right away.

Does the pidfile prevent duplicate beats?

Only on a single machine. Run one beat replica to prevent duplicates across servers.