Export Large CSV Files From Django Without Timeouts

Problem statement

An admin action or report view that exports orders to CSV works for a thousand rows and fails for half a million: the worker builds the whole file in memory, Gunicorn kills it at its timeout, Cloudflare returns a 524 after 100 seconds, or the OOM killer takes the worker down. Large exports need either streaming (send rows as they are read, with constant memory) or background generation (build the file in a task, store it, and give the user a link). Which one fits depends on size and how long the query takes.

1. For exports that finish in under a minute: StreamingHttpResponse with a generator over queryset.values_list(...).iterator(chunk_size=2000).
2. Use csv.writer with a pseudo-buffer that returns each row's text.
3. Send X-Accel-Buffering: no so Nginx streams rather than buffers.
4. For huge or slow exports: a Celery task writes the CSV to object storage and emails or shows a signed download link.
5. Add a UTF-8 BOM or offer an Excel format if users open files in Excel.
6. Restrict who can export and log exports; they are bulk data access.

Step-by-step solution

1. Stream rows with constant memory

import csv
from django.http import StreamingHttpResponse

class Echo:
    def write(self, value):
        return value

def export_orders(request):
    qs = (Order.objects.filter(account=request.user.account)
          .order_by("pk")
          .values_list("pk", "created_at", "customer__email", "status", "total"))
    writer = csv.writer(Echo())

    def rows():
        yield ""                                   # BOM for Excel
        yield writer.writerow(["id", "created_at", "email", "status", "total"])
        for row in qs.iterator(chunk_size=2000):
            yield writer.writerow(row)

    resp = StreamingHttpResponse(rows(), content_type="text/csv; charset=utf-8")
    resp["Content-Disposition"] = 'attachment; filename="orders.csv"'
    resp["X-Accel-Buffering"] = "no"
    return resp

values_list skips building model instances, and .iterator() uses a server-side cursor on PostgreSQL so rows arrive in chunks instead of all at once. Memory stays flat regardless of row count.

2. Mind the timeouts

Streaming sends the first bytes immediately, which satisfies proxies that wait for a response to start (Cloudflare's 100-second limit applies to the first byte). Gunicorn's sync worker timeout, however, still applies to the whole request; a 10-minute stream needs a longer timeout for that service, or an async worker. If the database query itself takes minutes before the first row, streaming does not help: use a background job.

3. Background exports for very large files

@shared_task(time_limit=3600)
def build_export(export_id):
    exp = Export.objects.get(pk=export_id)
    buf = io.StringIO()
    writer = csv.writer(buf)
    writer.writerow(["id", "created_at", "email", "status", "total"])
    with default_storage.open(f"exports/{exp.pk}.csv", "w") as f:
        for row in exp.queryset().values_list(...).iterator(chunk_size=5000):
            writer.writerow(row)
            if buf.tell() > 1_000_000:
                f.write(buf.getvalue()); buf.seek(0); buf.truncate()
        f.write(buf.getvalue())
    exp.status = "ready"; exp.save(update_fields=["status"])
    notify_export_ready.delay(exp.pk)

The user clicks "Export", gets a confirmation, and later receives an email or sees a link to a short-lived signed URL. This removes all request timeouts from the equation and lets you add progress tracking.

4. Excel compatibility

Excel expects a BOM to detect UTF-8 and may reformat values like leading-zero codes or long numbers. Either prefix the BOM (as above), document that users should import rather than double-click, or offer XLSX via openpyxl in write-only mode for streaming-friendly generation.

5. Security

Exports move a lot of data at once. Check permissions on the queryset, rate-limit export endpoints, log who exported what, and guard against CSV injection by prefixing cells that start with =, +, -, or @ with a single quote when users will open files in spreadsheets.

Verification check: export a production-sized dataset on staging and watch worker memory stay flat (ps -o rss), confirm the download starts within seconds and completes, and check that the file opens correctly in a spreadsheet with accented characters intact.

Explanation

A normal HttpResponse holds the entire body in memory before sending, and evaluating a queryset caches every model instance, so memory grows with the dataset. StreamingHttpResponse consumes a generator, sending each chunk as it is produced, and .iterator() with values_list fetches rows in batches without building model objects, so memory stays constant. Streaming solves memory and first-byte timeouts but not total request duration, which is why very large or slow exports belong in background jobs that store the file and notify the user.

Edge cases / notes

• PgBouncer in transaction mode breaks server-side cursors; set DISABLE_SERVER_SIDE_CURSORS = True, and .iterator() then fetches in chunks client-side.
• Admin actions can trigger background exports instead of returning files directly.
• Compression. Gzip at the proxy reduces transfer size; for stored files, offer .csv.gz.
• Large JOINs in exports benefit from indexes on filter and order columns.

Next steps

• If a proxy cuts the export off, see Fix Django StreamingHttpResponse and SSE Buffered by Nginx.
• For Cloudflare's limit, read Fix Cloudflare 524 Timeouts on Slow Django Requests.

FAQ

How do I export millions of rows to CSV in Django?

Stream with StreamingHttpResponse and values_list().iterator(), or generate the file in a background task for very large exports.

Why does my CSV export use so much memory?

The queryset and response are fully built in memory. Stream rows instead.

Does streaming avoid Gunicorn timeouts?

No. Gunicorn's timeout applies to the whole request; long exports need a longer timeout or a background job.

How do I make Excel open UTF-8 CSV correctly?

Prefix the file with a UTF-8 BOM, or generate XLSX.