Handle Large and Resumable File Uploads in Django

Problem statement

Users upload videos, datasets, or design files of hundreds of megabytes or several gigabytes. Through a normal Django form, the upload ties up a Gunicorn worker for minutes, hits proxy timeouts and body size limits, and fails completely if the connection drops at 90 percent, forcing the user to start over. Large uploads need to bypass Django's request cycle and be resumable.

1. Upload directly from the browser to S3 (or compatible storage) using multipart presigned URLs generated by Django.
2. For resumable uploads with a standard client, use the tus protocol with a tus server (tusd) or Uppy with S3 multipart.
3. Keep Django out of the data path: it creates the upload, signs parts, and completes it.
4. Process files (virus scan, thumbnails, transcoding, parsing) in Celery after the upload completes.
5. If uploads must go through Django, raise Nginx client_max_body_size, stream to disk, and use async or threaded workers.

Step-by-step solution

1. Create a multipart upload from Django

import boto3
from django.http import JsonResponse

s3 = boto3.client("s3", region_name="eu-central-1")

@login_required
def start_upload(request):
    key = f"uploads/{request.user.pk}/{uuid.uuid4()}/{get_valid_filename(request.POST['filename'])}"
    mp = s3.create_multipart_upload(Bucket=settings.UPLOAD_BUCKET, Key=key,
                                    ContentType=request.POST["content_type"])
    Upload.objects.create(user=request.user, key=key, upload_id=mp["UploadId"], status="uploading")
    return JsonResponse({"key": key, "upload_id": mp["UploadId"]})

@login_required
def sign_part(request):
    upload = get_object_or_404(Upload, upload_id=request.GET["upload_id"], user=request.user)
    url = s3.generate_presigned_url("upload_part", ExpiresIn=3600, Params={
        "Bucket": settings.UPLOAD_BUCKET, "Key": upload.key,
        "UploadId": upload.upload_id, "PartNumber": int(request.GET["part"])})
    return JsonResponse({"url": url})

The browser splits the file into parts (5 MB minimum, except the last), PUTs each part to its presigned URL, and records the ETag returned for each.

2. Complete the upload

@login_required
def complete_upload(request):
    upload = get_object_or_404(Upload, upload_id=request.POST["upload_id"], user=request.user)
    parts = json.loads(request.POST["parts"])     # [{"PartNumber": 1, "ETag": "..."}, ...]
    s3.complete_multipart_upload(Bucket=settings.UPLOAD_BUCKET, Key=upload.key,
                                 UploadId=upload.upload_id, MultipartUpload={"Parts": parts})
    upload.status = "uploaded"
    upload.save(update_fields=["status"])
    transaction.on_commit(lambda: process_upload.delay(upload.pk))
    return JsonResponse({"ok": True})

If the connection drops, the client asks S3 which parts exist (list_parts, signed by Django) and uploads only the missing ones. Uppy's AWS S3 multipart plugin implements this client logic.

3. Or use tus

tus is an open protocol for resumable uploads. A tus server (tusd) stores uploads in S3 or on disk and calls hooks when an upload finishes. Django authenticates the upload through a pre-create hook and records the result in a post-finish hook. Clients (Uppy, tus-js-client, mobile SDKs) handle chunking and resuming by offset. This is a good fit when you need a standard protocol across web and mobile clients.

4. Process after upload

Never trust the client's file type or size. In process_upload, check the object's actual size with head_object, inspect content (magic bytes, python-magic), run antivirus if required, generate derivatives, and only then mark the upload as available. Abort and clean up stale multipart uploads with an S3 lifecycle rule (AbortIncompleteMultipartUpload after a day or two), or you pay for orphaned parts.

5. If uploads must go through Django

Sometimes storage must stay behind Django. Then:

• Nginx: client_max_body_size 2g;, proxy_request_buffering on; (Nginx buffers the body so Gunicorn is not tied up during the upload), and sufficient client_body_timeout.
• Django: keep FILE_UPLOAD_MAX_MEMORY_SIZE small so files spool to temporary files on disk; make sure FILE_UPLOAD_TEMP_DIR has space.
• Workers: use gthread or async workers for upload endpoints, and long enough timeouts.

Explanation

Multipart upload splits an object into independently uploaded parts that storage assembles on completion. Each part can be retried alone, which makes uploads resumable and parallelizable. Presigned URLs let the browser upload parts directly with credentials scoped to one part and one object, so Django never handles the bytes. The tus protocol achieves resumability differently, by tracking a byte offset on a tus server. Either way, Gunicorn workers stay free for normal requests.

Edge cases / notes

• CORS. The bucket needs CORS rules allowing PUT from your origin and exposing the ETag header to the browser.
• Part limits. S3 allows up to 10,000 parts; choose part sizes so the largest expected file fits.
• S3-compatible stores. R2, B2, MinIO, and Spaces support multipart uploads with small differences; test the full flow.

Next steps

• Start simpler with Upload Files Directly to S3 From Django.
• If uploads still go through Nginx, see Fix Nginx 413 for Django Uploads.
• Validate uploads with the Django File Upload Security Checklist.

FAQ

Why not upload large files through a Django form?

The upload occupies a worker for its whole duration, hits proxy limits, and cannot resume after a dropped connection.

What is tus?

An open protocol for resumable uploads, with servers like tusd and clients like Uppy and tus-js-client.

How do I resume an S3 multipart upload?

List the parts already uploaded and upload only the missing ones, then complete the upload.

How do I clean up abandoned uploads?

Add an S3 lifecycle rule that aborts incomplete multipart uploads after a day or two.