Handling Large File Size Limits #

A 900 MB video does not fail because “the file is too big” — it fails at the first hop whose request-body ceiling is lower than the file, and that hop is almost never the one you configured. Getting large uploads to work is therefore two jobs: enumerating every ceiling on the path from the <input type="file"> to the bucket, and then sizing your transfer so that no single HTTP request ever approaches the lowest one.

This topic sits inside upload fundamentals and browser APIs and is the sizing layer of the stack: it decides how many bytes go in a request before any of the other machinery matters. Once you have a plan, best practices for handling 500MB file uploads builds the concrete pipeline end to end, and multipart versus single PUT for files under 100MB argues the other side for anything mid-sized.

Prerequisites #

• [ ] Node 20+ with @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner installed
• [ ] A bucket you can CreateMultipartUpload against, plus s3:AbortMultipartUpload in the same policy
• [ ] Shell access to whatever terminates TLS — you cannot size parts against a ceiling you cannot read
• [ ] Bucket CORS with PUT allowed and ETag in ExposeHeaders, per fixing CORS preflight errors on S3 uploads
• [ ] TypeScript with "lib": ["DOM", "ES2022"] if you are typing the browser half
• [ ] A test file of a genuinely awkward size — 4.7 GB is better than 1 GB, because it crosses two ceilings at once

How it works #

Where the ceilings actually live #

There is no single “upload limit”. There is a chain of independent limits, each enforced by a different process, each with its own default, and the request dies at the first one it exceeds. The lowest ceiling in a typical managed stack is not the one people expect: nginx ships with client_max_body_size 1m, so an unconfigured reverse proxy rejects anything over a megabyte before your handler is even invoked.

Two of those bars are not configuration. Cloudflare’s proxied-request body cap is a plan attribute — 100 MB on Free and Pro, 200 MB on Business, 500 MB by default on Enterprise — and no header, page rule or Worker changes it, so files above it must leave the proxied path entirely. API Gateway’s 10 MB payload limit is likewise fixed for both REST and HTTP APIs. The rest are yours to raise, and raising nginx and Cloudflare upload size limits walks the exact directives; the point here is that raising them is the wrong first instinct. A 4 GB file that flows through your API is 4 GB of socket time, connection slots and disk spool on a box that exists to serve JSON.

Why the browser is rarely the constraint #

A File object is a handle to bytes on disk, not the bytes themselves. file.slice(start, end) returns another handle with an offset and a length; it copies nothing, which is why slicing a 12 GB file 1,500 times costs microseconds and roughly zero heap. The mechanics of that are covered in slicing large files with Blob.slice.

The browser only becomes the bottleneck when you force materialisation. await file.arrayBuffer() on a 3 GB file asks V8 for a 3 GB contiguous allocation; on 64-bit Chrome you get RangeError: Array buffer allocation failed, and on iOS Safari you usually get no error at all because the tab is killed by the OS memory monitor first. Base64 makes it worse by a third before you even start, which is the whole argument in base64 versus binary encoding. Pass the Blob directly to fetch and the network stack reads it from disk in small reads; heap stays flat regardless of file size.

The real browser-side constraints are subtler. Over HTTP/1.1 a Chromium origin gets six concurrent sockets, so a concurrency setting above six buys nothing. Over HTTP/2 you get one connection with multiplexed streams sharing a flow-control window, so raising concurrency past about eight mostly adds head-of-line contention. And a Blob handed to fetch is a snapshot of a disk location, not a copy — if the user moves or truncates the file mid-upload, later parts fail with NotReadableError: The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.

The multipart contract #

S3 multipart is a three-call protocol with a strict set of invariants worth memorising, because every one of them shows up as a production error eventually:

• A part is numbered 1 to 10,000 inclusive. 10,000 is a hard ceiling, not a quota you can raise.
• Every part except the final one must be at least 5 MiB. Undersized parts are accepted at UploadPart time and rejected at completion with EntityTooSmall: Your proposed upload is smaller than the minimum allowed size.
• A single part may be up to 5 GiB, and a completed object up to 5 TiB.
• Parts may arrive in any order and be re-uploaded freely; the last write to a part number wins.
• Nothing is visible in the bucket until CompleteMultipartUpload, and the parts are billed as storage the entire time they sit there.

The consequence of the 10,000-part ceiling is a formula, not a preference. Your minimum viable part size is fileSize / 10000, rounded up. At the AWS CLI’s default 8 MiB chunk you can only carry 78 GiB; above that the CLI silently doubles the chunk size, and if you hand-rolled the client instead, you get InvalidArgument: Part number must be an integer between 1 and 10000, inclusive somewhere around part 10,001, three hours into a transfer.

Choosing between one PUT, multipart and resumable #

Multipart is not free. It costs you a part manifest, an orphan-cleanup policy, an extra round trip at each end, and a completion call that can fail after every byte is safely stored. For most files the simpler thing is correct.

The right-hand branches imply persistence, because a multipart upload that survives a page reload needs the UploadId and the accumulated part list to survive with it — that is the subject of resumable upload state machines, and it is the main hidden cost of choosing multipart.

Step-by-step implementation #

1. Turn a file size into a part plan #

Do the arithmetic once, in a pure function, and unit-test it. Every bug in this area is a bug in this function.

// src/upload/plan.ts
export type Strategy = "single-put" | "multipart";

export interface UploadPlan {
  strategy: Strategy;
  partSize: number;
  partCount: number;
}

const MIB = 1024 * 1024;
const MIN_PART = 5 * MIB;               // S3 minimum for every part but the last
const MAX_PARTS = 10_000;               // hard protocol ceiling
const MAX_PART = 5 * 1024 * MIB;        // 5 GiB per part
const SINGLE_PUT_MAX = 100 * MIB;       // our policy, not S3's

export function planUpload(fileSize: number, preferredPart = 8 * MIB): UploadPlan {
  if (!Number.isInteger(fileSize) || fileSize <= 0) {
    throw new RangeError(`refusing to plan an upload of ${fileSize} bytes`);
  }
  if (fileSize > MAX_PART * MAX_PARTS) {
    throw new RangeError(`${fileSize} bytes exceeds the 5 TiB object ceiling`);
  }
  if (fileSize <= SINGLE_PUT_MAX) {
    return { strategy: "single-put", partSize: fileSize, partCount: 1 };
  }

  // Double the part size until the part count fits under the protocol ceiling.
  let partSize = Math.max(MIN_PART, preferredPart);
  while (Math.ceil(fileSize / partSize) > MAX_PARTS) {
    partSize *= 2;
  }
  return { strategy: "multipart", partSize, partCount: Math.ceil(fileSize / partSize) };
}

Run it against the sizes that actually break things:

planUpload(42 * 1024 ** 2)  → { strategy: 'single-put', partSize: 44040192, partCount: 1 }
planUpload(4 * 1024 ** 3)   → { strategy: 'multipart', partSize: 8388608,   partCount: 512 }
planUpload(2 * 1024 ** 4)   → { strategy: 'multipart', partSize: 268435456, partCount: 8192 }

Note the third line: a 2 TiB file quietly became a 256 MiB part size. That number now dictates your client memory budget, and nobody chose it — the ceiling did.

2. Issue the upload and its presigned part URLs #

The server never sees payload. It validates the plan, opens the multipart upload, and signs one URL per part. Signing is a local HMAC operation with no network call, so a few hundred at a time is cheap; a few thousand is not, which is why the batching below matters for very large files.

// src/server/uploads.ts
import {
  S3Client,
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
  AbortMultipartUploadCommand
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { randomUUID } from "node:crypto";

const s3 = new S3Client({ region: process.env.AWS_REGION });
const BUCKET = process.env.S3_BUCKET as string;
const MAX_BYTES = 5 * 1024 ** 3;        // policy ceiling for this endpoint
const SIGN_BATCH = 100;                 // URLs signed per response

export interface CreateInput {
  filename: string;
  size: number;
  contentType: string;
  partSize: number;
  partCount: number;
}

export interface CreateOutput {
  key: string;
  uploadId: string;
  parts: Array<{ partNumber: number; url: string }>;
  moreParts: boolean;
}

function reject(status: number, message: string): never {
  throw Object.assign(new Error(message), { status });
}

export async function createMultipart(input: CreateInput): Promise<CreateOutput> {
  if (input.size > MAX_BYTES) reject(413, `file exceeds the ${MAX_BYTES} byte limit`);
  if (input.partCount < 1 || input.partCount > 10_000) reject(400, "part count out of range");
  if (input.partCount > 1 && input.partSize < 5 * 1024 * 1024) reject(400, "part size below 5 MiB");
  if (input.partSize * input.partCount < input.size) reject(400, "part plan cannot cover the file");

  const key = `uploads/${new Date().toISOString().slice(0, 10)}/${randomUUID()}`;
  const created = await s3.send(new CreateMultipartUploadCommand({
    Bucket: BUCKET,
    Key: key,
    ContentType: input.contentType,
    Metadata: { "original-name": encodeURIComponent(input.filename) }
  }));

  const uploadId = created.UploadId as string;
  const batch = Math.min(input.partCount, SIGN_BATCH);
  const parts = await Promise.all(
    Array.from({ length: batch }, (_value, i) =>
      getSignedUrl(
        s3,
        new UploadPartCommand({ Bucket: BUCKET, Key: key, UploadId: uploadId, PartNumber: i + 1 }),
        { expiresIn: 3600 }
      ).then((url) => ({ partNumber: i + 1, url }))
    )
  );

  return { key, uploadId, parts, moreParts: input.partCount > batch };
}

A successful call logs one CreateMultipartUpload and nothing else:

POST /api/uploads 201 18ms
  key=uploads/2026-07-26/8f2c0d1a-4e57-4c9a-9a2f-1d3b6e0f77aa
  uploadId=2~Yz9Qx1kJb0nT8pQ  parts=100/512  moreParts=true

expiresIn: 3600 is a judgement call, not a default. SigV4 allows up to 604,800 seconds (7 days), but only when signing with long-lived IAM user credentials; on a task role the URL dies with the STS session regardless of what you asked for. One hour comfortably covers 100 parts on a domestic connection and keeps a leaked URL close to worthless. If you need the full picture on signing, S3 presigned URL workflows covers the credential side properly.

3. Upload parts with bounded concurrency #

The browser half is a worker pool over a queue. Four workers is the right starting point: enough to fill a 100 Mbit link, few enough that peak heap stays modest and HTTP/1.1’s six-socket limit is never the binding constraint.

// src/upload/multipart-client.ts
import type { UploadPlan } from "./plan.js";

interface PartUrl { partNumber: number; url: string }
interface CreateResponse { key: string; uploadId: string; parts: PartUrl[]; moreParts: boolean }

const PART_TIMEOUT_MS = 120_000;

async function putPart(url: string, body: Blob, signal: AbortSignal): Promise<string> {
  const timeout = new AbortController();
  const timer = setTimeout(() => timeout.abort(new DOMException("part timed out", "TimeoutError")), PART_TIMEOUT_MS);
  const composite = AbortSignal.any([signal, timeout.signal]);
  try {
    const res = await fetch(url, { method: "PUT", body, signal: composite });
    if (!res.ok) {
      throw new Error(`part rejected: HTTP ${res.status} ${(await res.text()).slice(0, 200)}`);
    }
    const etag = res.headers.get("etag");
    if (!etag) {
      throw new Error("no ETag readable — add ETag to the bucket CORS ExposeHeaders list");
    }
    return etag;
  } finally {
    clearTimeout(timer);
  }
}

export async function uploadInParts(
  file: File,
  plan: UploadPlan,
  session: CreateResponse,
  onProgress: (bytesSent: number) => void,
  signal: AbortSignal = new AbortController().signal,
  concurrency = 4
): Promise<Array<{ ETag: string; PartNumber: number }>> {
  const etags = new Array<string>(plan.partCount);
  const queue = [...session.parts];
  let bytesSent = 0;

  async function worker(): Promise<void> {
    for (let job = queue.shift(); job !== undefined; job = queue.shift()) {
      const start = (job.partNumber - 1) * plan.partSize;
      const slice = file.slice(start, Math.min(start + plan.partSize, file.size));
      etags[job.partNumber - 1] = await putPart(job.url, slice, signal);
      bytesSent += slice.size;
      onProgress(bytesSent);
    }
  }

  await Promise.all(Array.from({ length: concurrency }, () => worker()));
  return etags.map((ETag, i) => ({ ETag, PartNumber: i + 1 }));
}

AbortSignal.any() composes the caller’s cancel signal with the per-part timeout, so a user pressing Cancel and a stalled socket both unwind through the same path — the pattern is explored further in aborting uploads with AbortController and timeouts. Note the deliberate absence of retry logic here: a failed part should be requeued by the layer above with jittered backoff, and handling 413 and 507 errors during uploads explains which status codes are worth retrying at all.

4. Complete, or abort deliberately #

// src/server/complete.ts
import { CompleteMultipartUploadCommand, AbortMultipartUploadCommand } from "@aws-sdk/client-s3";
import { s3, BUCKET } from "./uploads.js";

export interface CompleteInput {
  key: string;
  uploadId: string;
  parts: Array<{ ETag: string; PartNumber: number }>;
}

export async function completeMultipart(input: CompleteInput): Promise<{ location: string }> {
  const ordered = [...input.parts].sort((a, b) => a.PartNumber - b.PartNumber);
  if (ordered.some((p, i) => p.PartNumber !== i + 1 || !p.ETag)) {
    throw Object.assign(new Error("part list has gaps or missing ETags"), { status: 400 });
  }
  try {
    const out = await s3.send(new CompleteMultipartUploadCommand({
      Bucket: BUCKET,
      Key: input.key,
      UploadId: input.uploadId,
      MultipartUpload: { Parts: ordered }
    }));
    return { location: out.Location as string };
  } catch (error) {
    await s3.send(new AbortMultipartUploadCommand({
      Bucket: BUCKET, Key: input.key, UploadId: input.uploadId
    }));
    throw error;
  }
}

The AbortMultipartUploadCommand in the catch block is not optional housekeeping. Without it, a failed completion leaves every uploaded part on the storage bill indefinitely, invisible in the console’s object listing.

5. Cap the body you do accept #

Some uploads must go through your server — small files, or anything needing synchronous processing. Content-Length is a claim from the client, so enforce the real thing while streaming.

// src/server/limit-body.ts
import type { IncomingMessage } from "node:http";

export class PayloadTooLarge extends Error {
  readonly status = 413;
  constructor(limit: number) {
    super(`request body exceeds ${limit} bytes`);
    this.name = "PayloadTooLarge";
  }
}

export function limitBody(req: IncomingMessage, limit: number): AsyncIterable<Buffer> {
  const declared = Number(req.headers["content-length"]);
  if (Number.isFinite(declared) && declared > limit) throw new PayloadTooLarge(limit);

  return (async function* stream() {
    let seen = 0;
    for await (const chunk of req) {
      seen += (chunk as Buffer).byteLength;
      if (seen > limit) {
        req.destroy();
        throw new PayloadTooLarge(limit);
      }
      yield chunk as Buffer;
    }
  })();
}

Rejecting on the declared length is a fast path; the counter inside the loop is the one that actually protects you, because a chunked request carries no Content-Length at all. Enforcing the same ceiling at the storage layer, so a leaked URL cannot be abused, is what enforcing upload size limits with S3 POST policies is for.

Configuration reference #

Client-side planner options:

Infrastructure ceilings you must check by hand:

Edge cases and gotchas #

The 413 that never reaches your logs #

If your application log has no entry for a failed upload, the rejection happened upstream. The distinguishing signal is how many bytes the client managed to send: a proxy that refuses on Content-Length closes the connection almost immediately, while your own handler only rejects after reading everything. The curl probe in the verification section below separates the two in one command; fixing the upstream case is the whole subject of raising nginx and Cloudflare upload size limits.

CORS silently nulls the ETag #

This is the single most common multipart failure in a browser. The PUT succeeds, S3 returns 200 with an ETag, and res.headers.get("etag") returns null — because a cross-origin response only exposes a safelisted set of headers unless the bucket’s CORS policy names the rest. Your part list then completes with empty ETags and S3 answers:

InvalidPart: One or more of the specified parts could not be found.
The part may not have been uploaded, or the specified entity tag
may not have matched the part's entity tag.

The fix is "ExposeHeaders": ["ETag"] on the bucket CORS rule, and it needs no code change on your side.

ETag is not a checksum on a multipart object #

For a single PUT, the ETag is the MD5 of the body. For a multipart object it is the MD5 of the concatenated part MD5s, followed by a hyphen and the part count — "9b2cf5c8e4d7a1f30e6b8c2d5a7e91f4-512". Comparing that to a hash of the original file will always fail, and the value changes if you re-upload the same bytes with a different part size. If you need real integrity, request ChecksumAlgorithm: "SHA256" at CreateMultipartUpload, send x-amz-checksum-sha256 per part, and compute the client side with Web Crypto in the browser — hashing incrementally per part rather than over the whole file.

Presigned URLs expire mid-upload #

A 40 GB upload on a 20 Mbit link takes four and a half hours. Sign all its part URLs at the start with a one-hour lifetime and the run dies around part 200 with:

AccessDenied
Request has expired
3600

Sign lazily in batches instead, refreshing as workers drain the queue. The neighbouring failure is RequestTimeTooSkewed: The difference between the request time and the current time is too large, which appears when a client’s clock is more than 15 minutes off — common on freshly imaged laptops and phones that have been in aeroplane mode. Neither is retryable without re-signing, so classify them separately from network errors in your retry policy; retrying fetch uploads with idempotency keys covers the classification pattern.

Peak memory is concurrency times part size #

@aws-sdk/lib-storage buffers queueSize × partSize before it writes anything to a socket, and a hand-rolled pool has the same shape whenever parts are read rather than streamed. The product grows quickly and nobody notices until a container hits its memory limit.

Passing a Blob to fetch avoids most of this in the browser, since the body is read from disk lazily; the numbers above are the worst case, which is what you get the moment anything in the chain calls arrayBuffer(). On the server, streaming straight through with the Streams API keeps the whole grid irrelevant.

Orphaned parts you are still paying for #

Every abandoned multipart upload — closed tab, crashed worker, failed completion — leaves its parts in the bucket. They do not appear in ListObjectsV2, they do not appear in the console’s object browser, and they are billed at full standard-storage rates. A single misbehaving client can accumulate terabytes before anyone notices the line item. Attach an AbortIncompleteMultipartUpload lifecycle rule with DaysAfterInitiation: 7 to every bucket that accepts uploads on the day you create it; expiring incomplete multipart uploads automatically has the exact rule.

Other providers, other numbers #

Nothing above transfers cleanly. Azure Block Blob uses blocks, not parts: up to 50,000 of them, each up to 4,000 MiB, and you must supply your own base64 block IDs of uniform length or you get InvalidBlockList. GCS resumable uploads use one session URI and byte-range PUTs where every chunk except the last must be a multiple of 256 KiB — a 5 MiB chunk works, a 5,000,000-byte chunk does not. The comparison in S3 vs GCS vs Azure Blob for media uploads lays the three models side by side.

Zero-byte and pathological files #

file.size === 0 is legal, common (users drag placeholder files), and breaks multipart: CreateMultipartUpload succeeds, no part is ever uploaded, and completion fails with MalformedXML because the part list is empty. Route empty files to a plain PutObject. Similarly, a file whose size changes between planning and upload — an actively-written log, a cloud-synced document — produces a final part shorter than expected and a Content-Length mismatch, so re-read file.size immediately before slicing the last part rather than trusting the plan.

Verification #

Start by proving where a large request dies. This single command distinguishes an edge rejection from an application rejection:

head -c 125829120 /dev/urandom > /tmp/120mb.bin

curl -sS -o /dev/null -X POST https://api.example.com/api/uploads/probe \
  -H 'content-type: application/octet-stream' \
  -H 'expect: 100-continue' \
  --data-binary @/tmp/120mb.bin \
  -w 'status=%{http_code} uploaded=%{size_upload} time=%{time_total}\n'

Read the result by the uploaded field, not the status code:

status=413 uploaded=0         time=0.31    # refused before the body — nginx or the CDN
status=413 uploaded=125829120 time=41.9    # your handler read it all, then rejected it
status=200 uploaded=125829120 time=39.4    # the chain accepts 120 MB end to end

Next, confirm the parts landed with the sizes you planned. Every part but the last must match exactly:

aws s3api list-parts \
  --bucket media-ingest-prod \
  --key "uploads/2026-07-26/8f2c0d1a-4e57-4c9a-9a2f-1d3b6e0f77aa" \
  --upload-id "$UPLOAD_ID" \
  --query 'Parts[].{part:PartNumber,bytes:Size}' --output table

Then check the bucket is not accumulating orphans. Anything older than your lifecycle window is a bug, not a slow user:

aws s3api list-multipart-uploads --bucket media-ingest-prod \
  --query 'Uploads[?Initiated<`2026-07-19`].[Key,UploadId,Initiated]' --output text

Finally, lock the planner’s invariants into tests so a future “let’s use 5 MiB parts everywhere” change fails in CI rather than at 3 a.m.:

// test/plan.test.ts
import { describe, it, expect } from "vitest";
import { planUpload } from "../src/upload/plan.js";

const MIB = 1024 * 1024;
const sizes = [1, 99 * MIB, 101 * MIB, 4 * 1024 ** 3, 2 * 1024 ** 4, 5 * 1024 ** 4];

describe("planUpload", () => {
  it("never exceeds the protocol ceilings", () => {
    for (const size of sizes) {
      const plan = planUpload(size);
      expect(plan.partCount).toBeLessThanOrEqual(10_000);
      expect(plan.partSize * plan.partCount).toBeGreaterThanOrEqual(size);
      if (plan.strategy === "multipart") {
        expect(plan.partSize).toBeGreaterThanOrEqual(5 * MIB);
      }
    }
  });

  it("rejects objects above 5 TiB", () => {
    expect(() => planUpload(6 * 1024 ** 4)).toThrow(/5 TiB/);
  });
});

In DevTools, the shape of a healthy run is unmistakable: exactly concurrency requests to the bucket host in flight at any moment, each with a Content-Length equal to partSize, each returning 200 with a visible ETag response header. If you see the ETag in the Network panel but null in JavaScript, the CORS ExposeHeaders list is the culprit and nothing else.

Frequently Asked Questions