Fast Bulk Inserts and Updates in Django

Problem statement

An import job saves 100,000 rows with Model.objects.create() in a loop and takes forty minutes, holding a Celery worker the whole time. A nightly job updates prices one save() at a time. Each call is a separate round trip to the database, often in its own transaction, plus signal handlers and per-row Python work. Django's bulk operations send many rows per query, which commonly makes such jobs 10-100 times faster, as long as you understand what they skip.

1. Inserts: Model.objects.bulk_create(objs, batch_size=1000) inside transaction.atomic().
2. Upserts: bulk_create(..., update_conflicts=True, unique_fields=[...], update_fields=[...]) (PostgreSQL, SQLite, MySQL/MariaDB).
3. Same change to many rows: queryset.update(price=F("price") * 1.05), a single SQL statement.
4. Different values per row: bulk_update(objs, ["price"], batch_size=1000).
5. Stream input in chunks so memory stays bounded; do not build a list of millions of objects.
6. For millions of rows, use PostgreSQL COPY through psycopg.

Step-by-step solution

1. Bulk inserts

from django.db import transaction

def import_products(rows):
    batch = []
    with transaction.atomic():
        for row in rows:                      # rows can be a generator from a CSV reader
            batch.append(Product(sku=row["sku"], name=row["name"], price=row["price"]))
            if len(batch) >= 1000:
                Product.objects.bulk_create(batch, batch_size=1000)
                batch.clear()
        if batch:
            Product.objects.bulk_create(batch, batch_size=1000)

On PostgreSQL, bulk_create returns objects with primary keys set. Batching keeps each statement a reasonable size and memory bounded.

2. Upserts

Product.objects.bulk_create(
    batch,
    update_conflicts=True,
    unique_fields=["sku"],
    update_fields=["name", "price", "updated_at"],
    batch_size=1000,
)

This generates INSERT ... ON CONFLICT (sku) DO UPDATE, inserting new SKUs and updating existing ones in one statement per batch, without first querying which exist.

3. Set-based updates

from django.db.models import F, Q
Product.objects.filter(category__slug="winter").update(price=F("price") * 0.9)
Order.objects.filter(status="pending", created_at__lt=cutoff).update(status="expired")

update() runs one SQL UPDATE regardless of row count. For very large updates on busy tables, update in batches by primary key range to keep transactions and locks short.

4. Different values per row

for p in products:
    p.price = new_prices[p.sku]
Product.objects.bulk_update(products, ["price"], batch_size=500)

bulk_update builds CASE WHEN expressions; it is much faster than individual saves but slower than update(), so use it only when values differ per row.

5. Know what bulk methods skip

• save() is not called: custom save logic and auto_now fields on bulk_update (set them yourself).
• pre_save/post_save signals do not fire; search indexing or cache invalidation hooked to signals must be triggered separately.
• Model validation (full_clean) does not run; validate input before building objects.
• Multi-table inheritance children are not supported by bulk_create.

6. COPY for very large imports

from django.db import connection

def copy_products(rows):
    with connection.cursor() as cur:
        with cur.connection.cursor().copy("COPY shop_product (sku, name, price) FROM STDIN") as copy:
            for r in rows:
                copy.write_row((r["sku"], r["name"], r["price"]))

With psycopg 3, COPY streams rows in PostgreSQL's bulk format and is the fastest load method. It bypasses the ORM completely, so handle defaults and sequences yourself.

Verification check: time the job before and after on a production-sized copy, compare row counts with the source, and confirm side effects that signals used to handle (search index, cache) are updated by the job.

Explanation

Each ORM call costs a network round trip, query parsing, and, in autocommit mode, a transaction commit that waits for the write-ahead log to flush to disk. With 100,000 rows, those fixed costs dominate the actual data work. Bulk operations amortise them: one INSERT with a thousand value tuples, one UPDATE for a whole set, one transaction for the job. COPY goes further with a streaming protocol designed for bulk loading. The price is bypassing per-object Python hooks, which is why bulk code must handle side effects explicitly.

Edge cases / notes

• Very large transactions hold locks and generate lots of WAL; for multi-million-row jobs, commit every batch or few batches.
• Replication lag spikes during big loads; throttle on replicas-sensitive systems.
• Sequences stay correct with bulk_create (it uses defaults); explicit IDs need a sequence reset.
• Celery import tasks should chunk work into separate tasks for retries and progress tracking.

Next steps

• After loading with explicit IDs, see Fix duplicate key value violates unique constraint After a Restore.
• For the reverse direction, Export Large CSV Files From Django Without Timeouts.

FAQ

How much faster is bulk_create than save()?

Often 10-100 times for large batches, depending on network latency and indexes.

Does bulk_create call save() or send signals?

No. Neither save() nor pre_save/post_save signals run.

How do I upsert in Django?

Use bulk_create with update_conflicts=True, unique_fields, and update_fields.

What batch_size should I use?

Around 500-2,000 rows per batch is a good starting point; measure for your row size.