Fix "SSL Error: Decryption Failed or Bad Record MAC" in Celery and Gunicorn

Problem statement

Celery tasks, or requests under Gunicorn with --preload, fail intermittently with:

django.db.utils.OperationalError: SSL error: decryption failed or bad record mac
django.db.utils.OperationalError: SSL SYSCALL error: EOF detected

The database is fine and TLS is configured correctly. These errors appear when two processes use the same TLS connection to PostgreSQL. That happens when a parent process opens a database connection and then forks children: each child inherits the socket and the TLS session state, and as soon as two of them send data, the encryption counters on the stream no longer match what the server expects. PostgreSQL drops the connection, and every process sharing it fails.

1. Do not run database queries in the parent before forking: at import time, in AppConfig.ready(), in Celery app setup, or in a Gunicorn --preload app.
2. Close inherited connections in each child after fork: Celery worker_process_init and Gunicorn post_fork hooks.
3. In Celery, keep the default prefork behavior that closes connections around tasks, and do not cache connections in module globals.
4. Remove --preload from Gunicorn if you cannot remove import-time queries.
5. Check custom code for connections opened at module level, such as raw psycopg.connect() calls.

Step-by-step solution

1. Find queries that run before fork

Typical sources:

# app/apps.py
class ShopConfig(AppConfig):
    def ready(self):
        from .models import Setting
        settings_cache = dict(Setting.objects.values_list("key", "value"))  # runs in parent

# module level
DEFAULT_CURRENCY = Currency.objects.get(code="EUR")   # runs at import

Celery's main process and Gunicorn with --preload import your code once, then fork workers. Any query at import or ready() time opens a connection in the parent, which every child inherits.

2. Close connections after fork

For Celery:

# config/celery.py
from celery.signals import worker_process_init
from django.db import connections

@worker_process_init.connect
def close_inherited_connections(**kwargs):
    for conn in connections.all():
        conn.close()

For Gunicorn:

# gunicorn.conf.py
def post_fork(server, worker):
    from django.db import connections
    for conn in connections.all():
        conn.close()

Closing in the child discards the inherited handle; Django opens a fresh connection on the next query. Closing the shared socket from a child can briefly disturb the parent's use of it, so the cleanest fix is still to avoid the parent query.

3. Move import-time queries to runtime

Replace module-level queries with lazy access:

from functools import lru_cache

@lru_cache(maxsize=1)
def default_currency():
    return Currency.objects.get(code="EUR")

The first call in each worker process runs the query on that worker's own connection. For values that change, use the cache framework with a timeout instead of lru_cache.

4. Reconsider preload

gunicorn --preload imports the app before forking to save memory and catch import errors early. It is safe only if import does not touch the database or other network services. If removing those calls is not practical, drop --preload; each worker then imports the app itself.

5. Check other forked clients

The same rule applies to Redis clients, HTTP connection pools, and gRPC channels created at import time. Redis usually tolerates it better, but TLS connections to Redis show similar corruption. Create clients lazily or in post-fork hooks.

Explanation

TLS records carry sequence numbers and message authentication codes that depend on every record sent before. When two processes write to the same TLS socket, each believes it is continuing the stream from the shared starting point, so their records are out of order from the server's view. The server cannot authenticate them and aborts with "bad record mac", or the connection is reset, which the client reports as EOF detected. Without TLS the same sharing still corrupts the PostgreSQL protocol, just with different errors.

Edge cases / notes

• Solo and threads pools. They do not fork worker processes, so connections cannot be shared this way; the problem is specific to the prefork pool.
• Beat. Beat with the database scheduler queries in its own process and does not fork; it is not affected.
• PgBouncer. A pooler does not fix shared client sockets; the corruption happens before PgBouncer sees the data.

Next steps

• For connections closed under long tasks, see Fix Django InterfaceError: Connection Already Closed.
• Use preload safely with Gunicorn Preload App for Django.
• For certificate problems, read Fix PostgreSQL SSL Connection Errors in Django.

FAQ

Is my TLS configuration wrong?

Usually not. The error comes from two processes sharing one TLS connection after a fork.

Why does it only happen sometimes?

Corruption occurs when two children use the inherited connection at around the same time, which depends on timing and load.

Does disabling SSL fix it?

It changes the error message but not the bug. Shared connections corrupt the protocol with or without TLS.

Do I need both the Celery and Gunicorn hooks?

Only for the servers you run with forking and pre-fork imports. Removing queries before fork makes both unnecessary.