Best Practices for Handling 500MB File Uploads #

Slice the file into 16 MB parts, PUT four of them at a time straight to object storage with presigned part URLs, record every returned ETag, and finish with a server-side complete call — that keeps peak browser memory near 64 MB and turns a network drop into a 16 MB retry instead of a 500 MB restart.

This article sits under handling large file size limits inside upload fundamentals and browser APIs. It is the concrete build for half-gigabyte payloads: the numbers, the orchestrator, and the failures you will actually hit in production.

When to use this approach #

• Your users upload single files between roughly 100 MB and a few gigabytes — video masters, RAW photo bundles, database dumps, CAD archives. Below 100 MB the arithmetic changes and a single request usually wins; see multipart vs single-PUT for files under 100MB.
• The upload has to survive a Wi-Fi handover, a sleeping laptop lid, or a tab crash without starting over.
• You can put the bytes in object storage directly. If regulation forces every byte through your own service, keep the same part loop but expect to also raise the proxy ceilings covered in raising Nginx and Cloudflare upload size limits.

Prerequisites #

1. Node 20+ with "type": "module", @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner v3.600 or newer.
2. A bucket whose CORS policy allows PUT, allows the content-type request header, and — critically — sets ExposeHeaders: ["ETag"].
3. Environment: AWS_REGION, UPLOAD_BUCKET, and a role permitting s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts on arn:aws:s3:::your-bucket/incoming/*.
4. A browser baseline of Chrome 116+, Safari 17.4+ or Firefox 124+ — that is where AbortSignal.any() landed.

Why 500 MB breaks the naive path #

The failure is almost never bandwidth. It is that the obvious APIs materialise the whole file in the JavaScript heap. FileReader.readAsArrayBuffer() on a 500 MB file allocates 500 MB of contiguous heap; readAsDataURL() allocates that and a Base64 string 4/3 the size, so peak residency lands near 1.17 GB. A mobile Safari tab is killed somewhere around 300–400 MB of JS heap, and desktop Chrome throws RangeError: Array buffer allocation failed well before a gigabyte.

file.slice(start, end) avoids all of it. A Blob returned by slice is a reference into the same on-disk backing store, not a copy — nothing is read until the fetch body is drained, and only the bytes in flight ever occupy memory. Four concurrent 16 MB parts is 64 MB of real allocation regardless of whether the file is 500 MB or 5 GB. The mechanics of that byte-range view are covered in slicing large files with Blob.slice.

Two more myths worth killing before you write code. Content-Length is a forbidden header name — assigning it in a fetch init is silently dropped, and the browser computes the real value from the body. And keepalive: true caps the total in-flight body at 65,536 bytes; attach it to a 16 MB part and Chrome rejects the call with a bare TypeError: Failed to fetch. Use AbortController and timeouts to control request lifetime instead.

Sizing parts and the concurrency window #

Two numbers decide everything: part size and how many parts are in flight. S3 fixes the boundaries — a non-final part must be at least 5 MiB, and one upload may not exceed 10,000 parts. For 500 MB those limits are generous, so pick on throughput instead.

Every part costs one TLS-warm round trip plus a signature verification, on the order of 60–150 ms of fixed overhead. At 5 MB you pay it 100 times; at 64 MB you pay it 8 times but a single failure re-sends 64 MB and progress updates crawl. The band that behaves well on both fibre and a train Wi-Fi is 8–16 MB, with concurrency 4.

Concurrency above 6 rarely helps. A browser allows six connections per origin on HTTP/1.1, and on a 100 Mbit link four parallel streams already saturate the pipe; the seventh request just queues while inflating your memory bill. If you need the progress bar to stay honest while parts land out of order, feed the byte counter into accurate time-remaining estimates rather than counting completed parts.

Implementation #

The server owns the credentials and the upload identity; the browser only ever sees short-lived URLs. Start with the API side — three operations: create, re-sign, complete.

// uploads.mjs — Node 20+, "type": "module"
import { randomUUID } from "node:crypto";
import {
  S3Client,
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
  ListPartsCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const s3 = new S3Client({ region: process.env.AWS_REGION });
const BUCKET = process.env.UPLOAD_BUCKET;
const PART_SIZE = 16 * 1024 * 1024; // 16 MiB
const MAX_BYTES = 2 * 1024 * 1024 * 1024; // hard ceiling: 2 GiB
const URL_TTL = 3600; // seconds — must outlive the slowest realistic upload

export async function signParts(key, uploadId, from, to) {
  const urls = [];
  for (let partNumber = from; partNumber <= to; partNumber++) {
    const command = new UploadPartCommand({
      Bucket: BUCKET,
      Key: key,
      UploadId: uploadId,
      PartNumber: partNumber,
    });
    urls.push(await getSignedUrl(s3, command, { expiresIn: URL_TTL }));
  }
  return urls;
}

export async function createUpload({ filename, size, contentType }) {
  if (!Number.isInteger(size) || size <= 0 || size > MAX_BYTES) {
    throw new Error(`refusing declared size ${size}`);
  }
  const key = `incoming/${new Date().toISOString().slice(0, 10)}/${randomUUID()}/${filename}`;
  const { UploadId } = await s3.send(
    new CreateMultipartUploadCommand({
      Bucket: BUCKET,
      Key: key,
      ContentType: contentType || "application/octet-stream",
    }),
  );
  const partCount = Math.ceil(size / PART_SIZE);
  return {
    key,
    uploadId: UploadId,
    partSize: PART_SIZE,
    partCount,
    urls: await signParts(key, UploadId, 1, partCount),
  };
}

export async function listUploadedParts(key, uploadId) {
  const parts = [];
  let marker;
  do {
    const page = await s3.send(
      new ListPartsCommand({
        Bucket: BUCKET,
        Key: key,
        UploadId: uploadId,
        PartNumberMarker: marker,
      }),
    );
    for (const p of page.Parts ?? []) {
      parts.push({ PartNumber: p.PartNumber, ETag: p.ETag, Size: p.Size });
    }
    marker = page.IsTruncated ? page.NextPartNumberMarker : undefined;
  } while (marker);
  return parts.sort((a, b) => a.PartNumber - b.PartNumber);
}

export async function completeUpload(key, uploadId, expectedParts) {
  const parts = await listUploadedParts(key, uploadId);
  if (parts.length !== expectedParts) {
    throw new Error(`have ${parts.length} of ${expectedParts} parts — not completing`);
  }
  const result = await s3.send(
    new CompleteMultipartUploadCommand({
      Bucket: BUCKET,
      Key: key,
      UploadId: uploadId,
      MultipartUpload: {
        Parts: parts.map(({ PartNumber, ETag }) => ({ PartNumber, ETag })),
      },
    }),
  );
  return { location: result.Location, etag: result.ETag, partCount: parts.length };
}

Deriving the completion list from ListParts rather than trusting the client’s array of ETags is the single change that makes resume cheap: the authoritative record of what arrived lives in S3, so a browser that lost its state can ask for it. The signing calls themselves are ordinary v4 signatures — the same machinery described in generating secure presigned URLs with AWS SDK v3.

Now the browser side. This orchestrator runs a fixed pool of workers over a shared part index, which is both simpler and more correct than racing an array of promises and splicing the winner out.

export interface UploadSession {
  key: string;
  uploadId: string;
  partSize: number;
  partCount: number;
  urls: string[];
}

export interface UploadOptions {
  concurrency?: number;
  perPartTimeoutMs?: number;
  maxRetries?: number;
  signal?: AbortSignal;
  onProgress?: (sentBytes: number, totalBytes: number) => void;
}

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

function persist(uploadId: string, done: Map<number, string>): void {
  sessionStorage.setItem(`mpu:${uploadId}`, JSON.stringify([...done]));
}

function restore(uploadId: string): Map<number, string> {
  const raw = sessionStorage.getItem(`mpu:${uploadId}`);
  return raw ? new Map(JSON.parse(raw) as [number, string][]) : new Map();
}

async function putPart(
  url: string,
  body: Blob,
  timeoutMs: number,
  outer?: AbortSignal,
): Promise<string> {
  const timeout = AbortSignal.timeout(timeoutMs);
  const signal = outer ? AbortSignal.any([outer, timeout]) : timeout;
  const res = await fetch(url, { method: "PUT", body, signal });
  if (!res.ok) {
    const detail = (await res.text().catch(() => "")).slice(0, 200);
    const err = new Error(`HTTP ${res.status} ${detail}`) as Error & { status: number };
    err.status = res.status;
    throw err;
  }
  const etag = res.headers.get("ETag");
  if (!etag) throw new Error("no ETag on the response — add ExposeHeaders: [\"ETag\"] to bucket CORS");
  return etag;
}

async function withRetry<T>(fn: () => Promise<T>, maxRetries: number, part: number): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (cause) {
      const status = (cause as { status?: number }).status;
      const fatal = status !== undefined && !RETRYABLE.has(status);
      if (fatal || attempt >= maxRetries) {
        throw new Error(`part ${part} gave up after ${attempt + 1} attempts: ${(cause as Error).message}`);
      }
      const ceiling = Math.min(30_000, 500 * 2 ** attempt);
      await new Promise((r) => setTimeout(r, ceiling * (0.5 + Math.random())));
    }
  }
}

export async function uploadLargeFile(
  file: File,
  session: UploadSession,
  options: UploadOptions = {},
): Promise<{ PartNumber: number; ETag: string }[]> {
  const {
    concurrency = 4,
    perPartTimeoutMs = 120_000,
    maxRetries = 5,
    signal,
    onProgress,
  } = options;

  const done = restore(session.uploadId);
  let sent = [...done.keys()].length * session.partSize;
  let nextPart = 1;

  async function worker(): Promise<void> {
    for (;;) {
      const partNumber = nextPart++;
      if (partNumber > session.partCount) return;
      if (done.has(partNumber)) continue;

      const start = (partNumber - 1) * session.partSize;
      const blob = file.slice(start, Math.min(start + session.partSize, file.size));
      const etag = await withRetry(
        () => putPart(session.urls[partNumber - 1], blob, perPartTimeoutMs, signal),
        maxRetries,
        partNumber,
      );

      done.set(partNumber, etag);
      persist(session.uploadId, done);
      sent += blob.size;
      onProgress?.(Math.min(sent, file.size), file.size);
    }
  }

  await Promise.all(Array.from({ length: concurrency }, () => worker()));
  return [...done.entries()]
    .map(([PartNumber, ETag]) => ({ PartNumber, ETag }))
    .sort((a, b) => a.PartNumber - b.PartNumber);
}

The parameters that matter #

• nextPart++ is the whole scheduler. JavaScript’s single-threaded event loop makes the increment atomic, so four workers can share it without a lock and no part is ever uploaded twice.
• AbortSignal.any([outer, timeout]) composes the caller’s cancel signal with a per-part deadline. AbortSignal.timeout(120_000) alone would strand a cancelled upload; the caller’s signal alone would let one wedged socket hang forever.
• perPartTimeoutMs must be generous: 16 MB on a 2 Mbit/s uplink takes 64 seconds. 120 s is the smallest safe default; anything near 30 s fails healthy uploads on hotel Wi-Fi.
• RETRYABLE deliberately excludes 403. An expired signature returns 403 SignatureDoesNotMatch and will return it forever — retrying is a waste of the user’s data allowance.
• persist() writes after each part, not at the end. It costs one small sessionStorage write per 16 MB, which is invisible next to the network cost, and it is what makes a crashed tab resumable.
• The retry delay is min(30s, 500ms × 2^attempt) × (0.5 + random()), i.e. full jitter. The reasoning behind that shape lives in implementing exponential backoff for failed chunks.

Configuration reference #

Configuration gotchas #

SignatureDoesNotMatch after the laptop sleeps. The response body reads <Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message>, and the real cause is usually an expired X-Amz-Expires window rather than a bad key. A 500 MB upload on a 3 Mbit/s uplink runs 22 minutes; the SDK’s 15-minute default guarantees failure. Set expiresIn: 3600 and re-sign the remaining parts on resume.

res.headers.get("ETag") returns null. The PUT succeeded with 200, but the bucket CORS policy has no ExposeHeaders, so the browser hides every response header except the CORS-safelisted seven. Add "ExposeHeaders": ["ETag"] to the rule; the wider preflight story is in fixing CORS preflight errors on S3 uploads.

EntityTooSmall at completion. Your proposed upload is smaller than the minimum allowed size means some non-final part fell under 5 MiB — almost always because the part size was computed from a stale file.size, or because the last part was uploaded under the wrong PartNumber and is therefore no longer last.

413 Content Too Large when you proxy instead. If you route parts through your own server, Nginx’s default client_max_body_size 1m rejects a 16 MB part before your handler runs. Raise it, and handle the client side of that response as described in handling 413 and 507 errors during uploads.

Edge cases #

The tab sleeps and the part URLs expire #

Mobile Safari suspends background tabs aggressively, and a laptop lid closes mid-upload every day. On return, ask the server what actually landed: ListParts is the ground truth, the client’s sessionStorage map is only a fast path. Re-sign from the lowest missing part number and hand the browser a fresh URL array before restarting the pool.

The file changes under the reference #

A File handle points at a path plus a modification time. If the user re-exports the video while the upload runs, the next fetch on a slice throws DOMException: NotReadableError — sometimes only for parts past the truncation point, which produces a corrupted object if you ignore it. Capture file.lastModified and file.size when the session is created, key the resume record on both, and treat NotReadableError as fatal rather than retryable.

Verifying integrity across a resume #

A multipart object’s ETag is not the MD5 of the file — it is a hash of the concatenated part hashes with a -32 suffix, so you cannot compare it with a local checksum. If integrity matters, hash the file separately with file checksums in the browser using Web Crypto and store the digest as object metadata for the post-upload job to confirm.

Duplicate sessions from an impatient user #

Two tabs, two clicks, two CreateMultipartUpload calls, two 500 MB objects. Key the create endpoint on a client-generated request identifier so a repeated call returns the existing session instead of a new one — the same discipline as retrying fetch uploads with idempotency keys. Abandoned uploads still hold billable storage until they are aborted, so pair it with a rule for expiring incomplete multipart uploads automatically.

Verification #

Prove the pool respects its bounds and the object assembled correctly. In the browser, watch the concurrency and the final part list:

const file = new File([new Uint8Array(524_288_000)], "master.mov", { type: "video/quicktime" });
const session: UploadSession = await (await fetch("/uploads", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ filename: file.name, size: file.size, contentType: file.type }),
})).json();

console.assert(session.partCount === 32, `expected 32 parts, got ${session.partCount}`);

const parts = await uploadLargeFile(file, session, {
  concurrency: 4,
  onProgress: (sent, total) => console.log(`${((sent / total) * 100).toFixed(1)}%`),
});

console.assert(parts.length === session.partCount, "every part must be acknowledged");
console.assert(parts.every((p, i) => p.PartNumber === i + 1), "part numbers must be dense and ordered");
console.log("uploaded", parts.length, "parts");

Then confirm from the storage side. The -32 suffix on the ETag is the part count, which is the cheapest proof that the object was assembled from exactly the parts you sent:

aws s3api head-object --bucket "$UPLOAD_BUCKET" --key "$KEY" \
  --query '{bytes:ContentLength,etag:ETag}'
# { "bytes": 524288000, "etag": "\"6b1f0c2d9e4a7f83c5b1e0d2a4f68c31-32\"" }

# Nothing should be left half-finished.
aws s3api list-multipart-uploads --bucket "$UPLOAD_BUCKET" --output text

If ContentLength matches file.size exactly and no stale upload identifiers remain, the run is clean.

Frequently Asked Questions