Upload Fundamentals & Browser APIs: Engineering Guide #

Every file upload is a negotiation between three things that do not trust each other: a browser that holds a handle to bytes it does not own, a network that will drop the connection at 87% for no stated reason, and an origin that must decide whether the bytes are safe before it lets them near a queue. This section maps that negotiation end to end — file acquisition in the DOM, payload preparation off the main thread, transport over HTTP, and ingestion at the origin or object store — and names the API, the failure signal, and the trade-off at each hand-off.

The material here is deliberately architectural. Each stage below states the problem, shows the smallest production-shaped implementation of it, and then hands you to the topic page where the mechanism is worked through in full. If you are debugging a specific error string, jump to Common failure modes; if you are choosing a transport for a new feature, start at the Decision matrix.

Architecture overview #

A production upload crosses four boundaries, and each boundary has a contract — a concrete artefact that the previous stage guarantees to the next. Acquisition hands over a File handle. Preparation hands over a Blob or a ReadableStream. Transport hands over an HTTP response with a durable identifier. Ingestion hands over a persisted object plus a row of metadata. Model these as four stages with explicit contracts rather than one monolithic submit() handler, because each fails independently and each needs a different recovery strategy.

Why the contracts matter for recovery #

The reason to name the artefacts is that recovery cost differs by an order of magnitude between them. Re-running transport is cheap — you already hold the Blob, and the browser can re-read it from disk. Re-running acquisition is expensive and often impossible: on iOS Safari a File obtained from the photo picker can become unreadable after the app is backgrounded, and re-reading it throws DOMException: The requested file could not be read, typically due to permission problems. That single fact drives most of the design decisions further down this page — hold the handle, not the bytes, but be ready for the handle to expire.

The same asymmetry applies at the ingestion boundary. If the origin has already persisted the object and your client never saw the response, retrying without an idempotency key produces a duplicate. A stable client-generated key turns the ambiguous “did it land?” case into a cheap lookup, which is why retrying fetch uploads with idempotency keys is a prerequisite for any retry policy rather than an optional extra.

Cross-cutting concerns #

Three concerns cut across all four boundaries. Designing them in at the start costs a day; retrofitting them after launch costs a quarter.

Security defaults #

Treat everything the browser tells you about a file as a hint. file.type is derived from the filename extension on Windows and from a small sniff table elsewhere; a file called payload.jpg containing a PHP script reports image/jpeg in every major browser. The client-side check exists to give the user a fast, friendly rejection, not to keep you safe — that is why the browser MIME string is unreliable and why the origin must re-derive the type from the leading bytes with server-side file validation.

Credentials never reach the bundle. When the browser talks to object storage directly, it does so with a short-lived signature that pins the method, the key prefix, the maximum content length and — with POST policies — the content type. Fifteen minutes of validity is a sensible default: long enough to survive a slow 4G handshake and a retry, short enough that a leaked URL in a shared screenshot is worthless by the time anyone tries it.

Cost #

The cost model of an upload has three terms: bytes on the wire, origin CPU seconds, and storage of things you never finished. Base64 attacks the first — it inflates every payload by exactly 4/3 plus padding, so a 48 MB video becomes 64 MB, and on a metered mobile plan that is a user-visible cost as well as an egress line item. Proxying through your own servers attacks the second: every byte is read, buffered and re-written, and a 1 vCPU container saturates at roughly 40–60 MB/s of pass-through traffic before it starts adding queueing latency. Abandoned multipart uploads attack the third — S3 bills for the parts of an upload that was never completed or aborted, invisibly, until someone runs ListMultipartUploads and finds 4 TB of orphans.

Performance #

The performance budget is dominated by two numbers: how much work happens on the main thread, and how many round trips you pay. Hashing a 500 MB file with crypto.subtle.digest on the main thread blocks it for roughly 1.5–3 seconds on a mid-range laptop and far longer on a phone — long enough that Chrome reports it as a long task and your Interaction to Next Paint collapses. Move it to a worker. Round trips are the other half: at a 660 ms RTT typical of a congested mobile link, a 1 MB part spends more time waiting than transferring, which is the entire argument for part sizes in the 5–25 MB range on anything but the flakiest network.

Acquiring the file: the File and Blob layer #

The first boundary is deceptively simple: an <input type="file"> change event hands you a FileList. The trap is what you do next. Calling await file.arrayBuffer() on a 700 MB video materialises 700 MB in the JavaScript heap, and Chrome’s per-tab heap ceiling on 64-bit desktop is around 4 GB but far lower on mobile — a mid-range Android device throws RangeError: Array buffer allocation failed well before 1 GB. A File is a handle to disk-backed data. Keep it that way for as long as possible, and use Blob.slice() to produce further lazy views rather than copies. The mechanics of that memory model, including when a slice does force a read, are worked through in File API & Blob Objects.

Validate synchronously and cheaply before you allocate anything. Size and extension checks cost microseconds and reject the common mistakes — the 4 GB ProRes master, the .dmg someone dragged in by accident — before you have opened a socket.

export interface AcquireOptions {
  maxBytes: number;
  allowedTypes: ReadonlySet<string>;
}

export type AcquireResult =
  | { ok: true; file: File }
  | { ok: false; reason: "too-large" | "wrong-type" | "empty" ; detail: string };

/** Cheap, synchronous gate. Runs before any allocation or network call. */
export function acquire(file: File, opts: AcquireOptions): AcquireResult {
  if (file.size === 0) {
    // Zero-byte files come from interrupted OS copies and from directories
    // dragged into Safari, which reports them as 0-byte files of type "".
    return { ok: false, reason: "empty", detail: `${file.name} is 0 bytes` };
  }
  if (file.size > opts.maxBytes) {
    const mb = (file.size / 1_048_576).toFixed(1);
    const cap = (opts.maxBytes / 1_048_576).toFixed(0);
    return { ok: false, reason: "too-large", detail: `${mb} MB exceeds the ${cap} MB cap` };
  }
  // file.type is advisory only — the origin re-derives it from the bytes.
  if (file.type !== "" && !opts.allowedTypes.has(file.type)) {
    return { ok: false, reason: "wrong-type", detail: `${file.type} is not accepted` };
  }
  return { ok: true, file };
}

const VIDEO: AcquireOptions = {
  maxBytes: 2 * 1024 * 1024 * 1024,
  allowedTypes: new Set(["video/mp4", "video/quicktime", "video/webm"]),
};

document.querySelector<HTMLInputElement>("#picker")?.addEventListener("change", (event) => {
  const input = event.currentTarget as HTMLInputElement;
  for (const file of Array.from(input.files ?? [])) {
    const result = acquire(file, VIDEO);
    if (!result.ok) console.warn(`rejected ${file.name}: ${result.detail}`);
    else console.info(`accepted ${file.name} (${file.size} bytes)`);
  }
});

Note the file.type !== "" guard. An empty type string is common and legitimate — Firefox reports "" for extensions it does not recognise, and every browser reports "" for a directory entry dropped onto a drop zone. Rejecting on an empty string turns a recoverable case into a dead end; let it through and let the byte-level check decide.

Establishing what the file actually is #

Extension-based and browser-reported types are guesses. The authoritative answer lives in the first few dozen bytes: FF D8 FF for JPEG, 89 50 4E 47 for PNG, and for MP4 the four bytes 66 74 79 70 (ftyp) at offset 4. Reading only that prefix costs one 64-byte disk read regardless of file size, because file.slice(0, 64).arrayBuffer() reads exactly the slice. This is the cheapest high-value check in the whole pipeline, and it belongs on the client purely as a fast user-facing signal — the definitive version runs at the origin. File Type Detection in the Browser covers the full signature table, the container formats that need offset-aware matching, and how the accept attribute interacts with it.

type Sniffed = { mime: string; ext: string } | null;

const SIGNATURES: ReadonlyArray<{ offset: number; bytes: number[]; mime: string; ext: string }> = [
  { offset: 0, bytes: [0xff, 0xd8, 0xff], mime: "image/jpeg", ext: "jpg" },
  { offset: 0, bytes: [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], mime: "image/png", ext: "png" },
  { offset: 0, bytes: [0x47, 0x49, 0x46, 0x38], mime: "image/gif", ext: "gif" },
  { offset: 8, bytes: [0x57, 0x45, 0x42, 0x50], mime: "image/webp", ext: "webp" },
  { offset: 4, bytes: [0x66, 0x74, 0x79, 0x70], mime: "video/mp4", ext: "mp4" },
  { offset: 0, bytes: [0x1a, 0x45, 0xdf, 0xa3], mime: "video/webm", ext: "webm" },
  { offset: 0, bytes: [0x25, 0x50, 0x44, 0x46], mime: "application/pdf", ext: "pdf" },
];

/** Reads only the first 32 bytes, whatever the file size. */
export async function sniff(file: File): Promise<Sniffed> {
  const head = new Uint8Array(await file.slice(0, 32).arrayBuffer());
  for (const sig of SIGNATURES) {
    if (head.length < sig.offset + sig.bytes.length) continue;
    let match = true;
    for (let i = 0; i < sig.bytes.length; i += 1) {
      if (head[sig.offset + i] !== sig.bytes[i]) { match = false; break; }
    }
    if (match) return { mime: sig.mime, ext: sig.ext };
  }
  return null;
}

export async function describeMismatch(file: File): Promise<string | null> {
  const actual = await sniff(file);
  if (!actual) return `${file.name}: unrecognised container`;
  if (file.type && file.type !== actual.mime) {
    return `${file.name}: browser said ${file.type}, bytes say ${actual.mime}`;
  }
  return null;
}

The WebP entry is the instructive one: its signature sits at offset 8, after the RIFF header and a four-byte length. Any sniffer that only compares prefixes at offset 0 will silently miss it and fall through to “unrecognised”, which is how a perfectly valid image ends up rejected in production.

Serialising the payload #

Once you know what the bytes are, you have to decide how they travel. There are exactly three shapes in practice: a multipart/form-data body carrying the file alongside metadata fields, a raw binary body with the metadata pushed into headers or the URL, and a Base64 string embedded in JSON. The third is almost always a mistake outside of tiny assets — it costs 33% more bytes and forces the origin to decode before it can even measure the payload.

Multipart is worth understanding at the byte level, because most multipart bugs are framing bugs rather than logic bugs. The body is a sequence of parts separated by a delimiter that must not occur anywhere inside the payload, each part carrying its own headers, then a blank line, then raw bytes.

The single most common multipart bug is setting Content-Type: multipart/form-data by hand. Doing so strips the auto-generated boundary parameter, and the origin responds with 400 Bad Request and a message like Multipart: Boundary not found. Pass a FormData to fetch with no Content-Type header at all and the browser writes the correct header for you. The generation and parsing rules, including how to handle non-ASCII filenames, are covered in Multipart Form Data Explained and its Node-side counterpart on parsing multipart/form-data in a Node server.

export interface UploadMeta {
  caption: string;
  capturedAt: string;
  checksum: string;
}

export async function postMultipart(
  endpoint: string,
  file: File,
  meta: UploadMeta,
  signal: AbortSignal,
): Promise<{ id: string }> {
  const body = new FormData();
  // Order matters: put small fields first so a streaming parser can read
  // metadata and reject early without buffering the file part.
  body.append("meta", new Blob([JSON.stringify(meta)], { type: "application/json" }));
  body.append("file", file, file.name);

  const response = await fetch(endpoint, {
    method: "POST",
    body,
    signal,
    // No Content-Type header. The browser adds it WITH the boundary parameter.
    headers: { "Idempotency-Key": meta.checksum },
  });

  if (!response.ok) {
    const text = await response.text();
    throw new Error(`upload failed: ${response.status} ${response.statusText} — ${text.slice(0, 200)}`);
  }
  return (await response.json()) as { id: string };
}

Appending the metadata part first is not cosmetic. A streaming parser reads parts in order, so metadata-first lets the origin authorise, size-check and reject before a single megabyte of video has been buffered. Metadata-last forces it to hold the whole file to reach the field it needs. The wire-format trade-offs against raw binary are quantified in Base64 vs Binary Encoding, and the mobile-specific version of the same arithmetic in optimizing payload size for mobile uploads.

Transport with fetch #

fetch is the transport for everything new. It gives you AbortController for cancellation and timeouts, proper Response objects, and — where the platform supports it — streaming request bodies. What it does not give you is upload progress: fetch has no equivalent of XMLHttpRequest.upload.onprogress, and the response promise resolves only after the response headers arrive. That single gap is why so much production code still reaches for XMLHttpRequest, and why the alternative is to wrap the body in a stream that counts bytes as they are pulled.

The timeout pattern is worth getting exactly right, because the naive version leaks. A setTimeout that is not cleared keeps a timer alive for the full duration even after a fast success, and on a page that uploads hundreds of thumbnails those timers accumulate. Clear it in finally, and distinguish a timeout abort from a user abort by checking which signal fired.

export interface TimedFetchInit extends RequestInit {
  timeoutMs: number;
  userSignal?: AbortSignal;
}

export class UploadTimeout extends Error {
  constructor(readonly ms: number) {
    super(`request exceeded the ${ms} ms budget`);
    this.name = "UploadTimeout";
  }
}

export async function timedFetch(url: string, init: TimedFetchInit): Promise<Response> {
  const { timeoutMs, userSignal, ...rest } = init;
  const timer = new AbortController();
  const id = setTimeout(() => timer.abort(new UploadTimeout(timeoutMs)), timeoutMs);
  // AbortSignal.any settles as soon as either the timeout or the user aborts.
  const signal = userSignal ? AbortSignal.any([timer.signal, userSignal]) : timer.signal;

  try {
    return await fetch(url, { ...rest, signal });
  } catch (error) {
    if (timer.signal.aborted && timer.signal.reason instanceof UploadTimeout) {
      throw timer.signal.reason; // retryable
    }
    if (userSignal?.aborted) {
      throw new DOMException("cancelled by the user", "AbortError"); // NOT retryable
    }
    throw error;
  } finally {
    clearTimeout(id);
  }
}

AbortSignal.any is the piece most implementations miss; before it shipped, combining a user cancellation with a timeout meant hand-wiring addEventListener("abort") on both and forgetting to remove the listener. The full lifecycle — including how to abort in-flight parts of a chunked transfer without orphaning the upload session — is in Modern Fetch API for Uploads and aborting uploads with AbortController and timeouts.

Streaming request bodies #

Passing a ReadableStream as the body of a fetch lets you send bytes you have not yet produced — the output of a WebCodecs encoder, an encrypted stream, or a progress-instrumented pass-through. It requires duplex: "half" in the init object, requires HTTP/2 or HTTP/3, and is not supported in Safari at the time of writing, so it must be feature-detected rather than assumed. Because it also lets you count bytes as they are consumed, it is the cleanest way to get upload progress out of fetch — the pattern is developed fully in Streams API for Uploads and specifically in tracking upload progress with a TransformStream.

/** True only where fetch will actually accept a stream body. */
export function supportsStreamingUpload(): boolean {
  let used = false;
  try {
    const probe = new Request("https://example.invalid", {
      method: "POST",
      body: new ReadableStream(),
      // @ts-expect-error duplex is not in older DOM lib typings
      duplex: "half",
      get headers() { used = true; return new Headers(); },
    });
    return used && !probe.headers.has("Content-Type");
  } catch {
    return false;
  }
}

export function countingStream(
  source: ReadableStream<Uint8Array>,
  onBytes: (total: number) => void,
): ReadableStream<Uint8Array> {
  let total = 0;
  return source.pipeThrough(
    new TransformStream<Uint8Array, Uint8Array>({
      transform(chunk, controller) {
        total += chunk.byteLength;
        onBytes(total);
        controller.enqueue(chunk);
      },
    }),
  );
}

The feature probe looks odd but it is the standard one: constructing a Request with a stream body only reads the headers getter on platforms that accept stream bodies, so the used flag is the signal. Guessing from the user-agent string instead will break the day Safari ships support.

Sizing the transfer #

Part size is the single most consequential number in a chunked upload, and the right value is not a constant — it is a function of bandwidth, round-trip time and failure rate. Too small and per-request overhead dominates: at a 660 ms RTT, a 1 MB part spends 62% of its wall-clock time in handshake and header exchange. Too large and every failure is expensive, because a dropped connection at 95% of a 50 MB part discards 47.5 MB of transferred data.

Read the curves as diminishing returns rather than absolute numbers. Going from 0.5 MB to 5 MB parts more than doubles effective throughput on both links. Going from 5 MB to 50 MB adds 19% on fibre and 29% on 4G — real, but bought at the price of a ten-fold larger retry unit. S3 also imposes hard rules: a minimum part size of 5 MiB for all parts except the last, a maximum of 5 GiB per part, and a ceiling of 10,000 parts per upload, which is what actually sets the floor for very large objects. Those constraints and the origin-side caps that interact with them are the subject of Handling Large File Size Limits; the proxy configuration that trips people up first is in raising Nginx and Cloudflare upload size limits.

const MIB = 1024 * 1024;
const S3_MIN_PART = 5 * MIB;
const S3_MAX_PARTS = 10_000;

export interface PartPlan {
  partSize: number;
  partCount: number;
  concurrency: number;
}

/**
 * Picks a part size that satisfies the 10,000-part ceiling, respects the
 * 5 MiB minimum, and shrinks on slow links so a retry costs less.
 * downlinkMbps comes from navigator.connection when available.
 */
export function planParts(fileSize: number, downlinkMbps: number): PartPlan {
  const preferred = downlinkMbps >= 20 ? 16 * MIB : downlinkMbps >= 5 ? 8 * MIB : 5 * MIB;
  const required = Math.ceil(fileSize / S3_MAX_PARTS);
  const partSize = Math.max(S3_MIN_PART, preferred, Math.ceil(required / MIB) * MIB);
  const partCount = Math.max(1, Math.ceil(fileSize / partSize));
  // More than six in-flight requests starves the rest of the app on HTTP/1.1
  // and gains nothing measurable on HTTP/2.
  const concurrency = Math.min(6, Math.max(1, Math.floor(downlinkMbps / 4)) || 1, partCount);
  return { partSize, partCount, concurrency };
}

export function readDownlink(): number {
  const conn = (navigator as Navigator & { connection?: { downlink?: number } }).connection;
  return typeof conn?.downlink === "number" && conn.downlink > 0 ? conn.downlink : 10;
}

navigator.connection.downlink is a rounded estimate and is absent in Safari and Firefox, hence the 10 Mbps fallback. Treat it as a starting guess and correct it from measured part durations after the first two parts complete — the initial value only needs to be close enough to avoid the pathological cases at either end.

Resilience: timeouts, retries and idempotency #

Retries are where naive implementations do real damage. A fixed 1-second retry loop across 400 clients that all failed on the same origin deploy produces a synchronised burst that keeps the origin down. Exponential backoff with full jitter is the fix, and the jitter is not optional — deterministic backoff merely moves the burst later.

The other half is knowing what is safe to retry. Retry on 408, 429, 500, 502, 503, 504 and on transport-level failures (TypeError: Failed to fetch, which is what a dropped connection looks like from fetch). Never retry 400, 403, 404, 413 or 422 — those are deterministic and will fail identically. And always honour Retry-After when the origin sends it, in either its seconds or its HTTP-date form.

const RETRYABLE_STATUS = new Set([408, 425, 429, 500, 502, 503, 504]);

export interface RetryPolicy {
  maxAttempts: number;
  baseMs: number;
  capMs: number;
}

const DEFAULT_POLICY: RetryPolicy = { maxAttempts: 5, baseMs: 500, capMs: 20_000 };

function backoffMs(attempt: number, policy: RetryPolicy): number {
  const ceiling = Math.min(policy.capMs, policy.baseMs * 2 ** attempt);
  return Math.random() * ceiling; // full jitter: uniform in [0, ceiling)
}

function retryAfterMs(response: Response): number | null {
  const header = response.headers.get("Retry-After");
  if (!header) return null;
  const seconds = Number(header);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
  const date = Date.parse(header);
  return Number.isNaN(date) ? null : Math.max(0, date - Date.now());
}

const sleep = (ms: number, signal: AbortSignal): Promise<void> =>
  new Promise((resolve, reject) => {
    const id = setTimeout(resolve, ms);
    signal.addEventListener("abort", () => { clearTimeout(id); reject(signal.reason); }, { once: true });
  });

export async function putPartWithRetry(
  url: string,
  part: Blob,
  idempotencyKey: string,
  signal: AbortSignal,
  policy: RetryPolicy = DEFAULT_POLICY,
): Promise<string> {
  let lastError = "unknown";
  for (let attempt = 0; attempt < policy.maxAttempts; attempt += 1) {
    if (attempt > 0) await sleep(backoffMs(attempt - 1, policy), signal);
    const response = await timedFetch(url, {
      method: "PUT",
      body: part,
      timeoutMs: 60_000,
      userSignal: signal,
      headers: { "Content-Type": "application/octet-stream", "Idempotency-Key": idempotencyKey },
    }).catch((error: unknown) => {
      lastError = error instanceof Error ? `${error.name}: ${error.message}` : String(error);
      return null;
    });

    if (response?.ok) {
      const etag = response.headers.get("ETag");
      if (!etag) throw new Error("no ETag returned — is ETag in Access-Control-Expose-Headers?");
      return etag.replaceAll('"', "");
    }
    if (response && !RETRYABLE_STATUS.has(response.status)) {
      throw new Error(`part ${idempotencyKey} failed permanently: HTTP ${response.status}`);
    }
    if (response) {
      lastError = `HTTP ${response.status}`;
      const hinted = retryAfterMs(response);
      if (hinted !== null) await sleep(hinted, signal);
    }
  }
  throw new Error(`part ${idempotencyKey} exhausted ${policy.maxAttempts} attempts (${lastError})`);
}

Two details earn their keep here. The ETag guard catches the most confusing CORS symptom in browser uploads: the PUT succeeds, the object lands in the bucket, and response.headers.get("ETag") returns null because the bucket’s CORS rule omitted ExposeHeaders: ["ETag"]. Without the header you cannot complete the multipart upload, and the failure looks like a storage bug rather than a configuration one — fixing CORS preflight errors on S3 uploads covers the whole rule set. The second detail is sleep accepting the abort signal, so cancelling an upload does not leave five timers waiting to fire retries against a session the user already abandoned. Backoff strategy, circuit breaking and the interaction with resumable sessions are expanded in Browser Timeout & Retry Logic and upload error recovery patterns.

Drag-and-drop acquisition #

Drag-and-drop is the primary acquisition path for desktop media tools, and it has a failure surface the file input does not. The dragover handler must call event.preventDefault(), or the browser treats the drop as a navigation and opens the file directly, replacing your application. Dropped items arrive as a DataTransferItemList rather than a FileList, and folders only yield their contents through the entry-traversal API. Build drop zones with the patterns in Drag-and-Drop File Uploads, which also covers keyboard-accessible fallbacks and the paste path.

export interface DroppedFile {
  file: File;
  path: string;
}

export async function collectDroppedFiles(dataTransfer: DataTransfer): Promise<DroppedFile[]> {
  const items = Array.from(dataTransfer.items).filter((i) => i.kind === "file");
  const out: DroppedFile[] = [];

  async function walk(entry: FileSystemEntry, prefix: string): Promise<void> {
    if (entry.isFile) {
      const file = await new Promise<File>((resolve, reject) =>
        (entry as FileSystemFileEntry).file(resolve, reject),
      );
      out.push({ file, path: prefix + entry.name });
    } else if (entry.isDirectory) {
      const reader = (entry as FileSystemDirectoryEntry).createReader();
      // readEntries returns at most 100 entries per call; loop until it returns none.
      for (;;) {
        const batch = await new Promise<FileSystemEntry[]>((resolve, reject) =>
          reader.readEntries(resolve, reject),
        );
        if (batch.length === 0) break;
        for (const child of batch) await walk(child, prefix + entry.name + "/");
      }
    }
  }

  for (const item of items) {
    const entry = item.webkitGetAsEntry();
    if (entry) await walk(entry, "");
  }
  return out;
}

export function attachDropZone(el: HTMLElement, onFiles: (files: DroppedFile[]) => void): void {
  el.addEventListener("dragover", (event) => {
    event.preventDefault(); // REQUIRED — otherwise the browser navigates to the file
    if (event.dataTransfer) event.dataTransfer.dropEffect = "copy";
  });
  el.addEventListener("drop", async (event) => {
    event.preventDefault();
    if (!event.dataTransfer) return;
    onFiles(await collectDroppedFiles(event.dataTransfer));
  });
}

The recursive readEntries loop is the part everyone gets wrong. Chrome returns at most 100 entries per call, so a single invocation on a 340-image folder silently returns the first 100 and no error at all. Users then report “it only uploaded some of them”, which is close to unfalsifiable without the loop in place. The folder-specific pitfalls, including Safari’s differing behaviour, are covered in handling dropped folders with the DataTransfer API.

Configuration reference #

These are the knobs that decide whether an upload pipeline holds up under real traffic. Defaults below are the ones worth starting from; each links to where the reasoning lives.

Decision matrix #

The transport decision reduces to two axes: how big the file is, and how likely the connection is to survive it. Everything else — metadata handling, progress reporting, origin cost — follows from the quadrant you land in.

The row that surprises people is the third: a ReadableStream body is the only transport that gives byte-accurate progress from fetch itself, but it forfeits automatic retry — a stream can be read once, so a retry needs a fresh stream from the original source. Budget for that when you choose it.

Common failure modes #

These are the signals you will actually see in a browser console or an origin log, with the cause and the fix.

The 507 case #

507 Insufficient Storage deserves its own note because it is usually misdiagnosed as a client bug. It appears when the origin writes uploads to a temporary directory that filled up — often because a previous batch of failed uploads left partial files behind and nothing cleaned them up. The client-side fix is to back off aggressively and surface a distinct message; the real fix is an eviction policy on the temp directory and a lifecycle rule on the bucket, which is why expiring incomplete multipart uploads automatically belongs in the initial build rather than a later hardening pass.

Silent truncation #

The nastiest class of failure produces no error at all. A readEntries loop that runs once, a Content-Length computed from a stale file.size after the user edited the file on disk, a proxy configured with proxy_request_buffering off in front of a handler that expects a buffered body — each of these produces a stored object that is smaller than the source with a 200 OK on the wire. The only defence is an end-to-end integrity check: hash the file on the client, send the digest, and have the origin compare it against what it actually persisted before it marks the upload complete.

Verification #

Assume nothing works until you have proved it at three levels: the wire, the browser, and the stored object.

On the wire. Confirm the proxy chain accepts your largest part before writing any client code. This posts 10 MiB of zeroes and prints only the status and timing:

head -c 10485760 /dev/zero > /tmp/part.bin
curl -sS -o /dev/null -w 'status=%{http_code} sent=%{size_upload} time=%{time_total}s\n' \
  -X PUT --data-binary @/tmp/part.bin \
  -H 'Content-Type: application/octet-stream' \
  https://uploads.example.com/v1/parts/probe

A 413 here means the proxy, not your handler. A 200 with sent= far below 10485760 means something truncated the body.

In the browser. Check the request the browser actually built, not the one you think you built. In DevTools the network entry for a FormData POST must show Content-Type: multipart/form-data; boundary=----WebKitFormBoundary… — if the boundary= parameter is missing, you set the header manually. For a presigned PUT, confirm the response’s Access-Control-Expose-Headers includes ETag.

Against the stored object. The definitive check compares the client-side digest with the persisted bytes:

export async function digestHex(blob: Blob): Promise<string> {
  const buffer = await crypto.subtle.digest("SHA-256", await blob.arrayBuffer());
  return Array.from(new Uint8Array(buffer))
    .map((b) => b.toString(16).padStart(2, "0"))
    .join("");
}

export async function verifyUpload(file: File, statusUrl: string): Promise<void> {
  // Hash the first and last 1 MiB plus the size: a cheap tripwire for truncation
  // that avoids reading a multi-gigabyte file twice.
  const head = await digestHex(file.slice(0, 1_048_576));
  const tail = await digestHex(file.slice(Math.max(0, file.size - 1_048_576)));

  const response = await fetch(statusUrl, { headers: { Accept: "application/json" } });
  const stored = (await response.json()) as { size: number; headSha256: string; tailSha256: string };

  if (stored.size !== file.size) {
    throw new Error(`size mismatch: sent ${file.size}, stored ${stored.size}`);
  }
  if (stored.headSha256 !== head || stored.tailSha256 !== tail) {
    throw new Error("content mismatch: the stored object is not the file that was selected");
  }
  console.info(`verified ${file.name}: ${file.size} bytes intact`);
}

Hashing only the first and last mebibyte is a deliberate trade-off. It catches every truncation and every off-by-one part-ordering bug at a fixed 2 MiB of reading, whereas a full-file digest of a 4 GB master costs 12–20 seconds. Use the full digest when you need integrity guarantees; use the tripwire when you need a fast post-upload sanity check on every transfer. Once verified, the object is ready for the ingestion side of the chain — server-side file validation, scanning, and post-upload media transcoding.

Frequently Asked Questions