Fix Redis "LOADING Redis Is Loading the Dataset in Memory" in Django

Problem statement

Right after Redis restarts, for a few seconds or a few minutes, Django and Celery log:

redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory

or LOADING Redis is loading the dataset in memory from other clients. Pages that touch the cache return 500 errors, Celery workers cannot fetch tasks, and health checks fail. Redis is up and accepting connections, but while it reads its RDB snapshot or AOF file from disk it refuses commands. The bigger the dataset, the longer this lasts.

1. Check progress: redis-cli INFO persistence | grep loading shows loading:1 and an ETA.
2. Make the Django cache fail softly: with django-redis set "IGNORE_EXCEPTIONS": True; with Django's built-in Redis cache, catch errors in a wrapper.
3. Let Celery retry broker connections on startup (broker_connection_retry_on_startup = True).
4. Separate cache and broker into different Redis instances; only the broker needs persistence.
5. Reduce load time: smaller datasets, faster disks, or no persistence for pure caches.

Step-by-step solution

1. Confirm Redis is loading

redis-cli INFO persistence | grep -E "loading|loading_eta|loading_loaded_perc"

loading:1 with a percentage confirms it. Redis logs DB loaded from disk: 42.318 seconds when finished. A multi-gigabyte AOF without a recent rewrite can take minutes.

2. Make the cache optional

A cache should speed pages up, not take them down. With django-redis:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://cache.internal:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "IGNORE_EXCEPTIONS": True,
            "SOCKET_CONNECT_TIMEOUT": 1,
            "SOCKET_TIMEOUT": 1,
        },
    }
}
DJANGO_REDIS_LOG_IGNORED_EXCEPTIONS = True

Cache reads then return None during loading, and the app falls back to the database. Django's built-in django.core.cache.backends.redis.RedisCache does not ignore exceptions, so wrap critical cache calls in try/except or use django-redis for this behavior.

3. Let Celery wait for the broker

Celery 5.3+ warns unless you decide how startup connection failures behave. Retrying is what you want:

CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True
CELERY_BROKER_CONNECTION_MAX_RETRIES = None   # keep retrying

Workers then wait until loading finishes instead of crashing and restarting in a loop. Producers in web requests that call .delay() will still fail during loading; wrap them or use transaction.on_commit() plus a retry.

4. Split cache and broker

One Redis for everything forces one persistence setting on data with different needs:

• Cache: disposable, no persistence needed. Restarts in under a second when empty.
• Celery broker and results: should survive restarts, so AOF or RDB is on, and loading takes time.
• Sessions: if stored in Redis, users log out when data is lost.

Run separate instances or at least separate processes, and configure persistence for each.

5. Shorten the load

# redis.conf for a cache-only instance
save ""
appendonly no

For persistent instances, keep AOF compact (auto-aof-rewrite-percentage 100), use Redis 7's multi-part AOF, put data on fast SSD storage, and set maxmemory so the dataset cannot grow without bound.

Explanation

On startup Redis reads its persistence file into memory before serving data, because serving from a half-loaded dataset would return wrong answers. To let clients know it is alive, it accepts connections and replies with the LOADING error to most commands. Replicas do the same after a full resynchronization with the primary. Managed services such as ElastiCache and Memorystore show it after node replacements or failovers when a replica loads a snapshot.

Edge cases / notes

• Health checks. A Kubernetes liveness probe that runs redis-cli ping during loading gets an error and restarts Redis, which starts loading again. Use a startup probe or a longer initial delay.
• Sentinel and Cluster. A newly promoted replica may still be loading if it resynced recently. Client retries cover the gap.
• Big keys. A few huge keys slow loading noticeably. redis-cli --bigkeys finds them.

Next steps

• Decide what needs persistence with Configure Redis Persistence for Django and Celery.
• Set up the cache with Set Up Redis Cache for Django in Production.
• Review the whole setup with the Django Redis Production Checklist.

FAQ

How long does Redis take to load?

Roughly proportional to dataset size and disk speed: seconds for hundreds of megabytes, minutes for many gigabytes or a large uncompacted AOF.

Is my data lost?

No. LOADING means Redis is restoring data from disk. Once finished, the persisted data is available.

Should a Django cache use persistence?

Usually not. A cache can be rebuilt from the database, and without persistence Redis restarts almost instantly.

Why does Celery crash during loading?

Workers that cannot reach the broker at startup exit unless connection retries are enabled. Turn on retry on startup.