Compressing Video in the Browser with WebCodecs #

Demux the MP4 yourself, feed its samples to a VideoDecoder, draw each VideoFrame onto an OffscreenCanvas at the target size, encode it with a VideoEncoder configured as avc1.640028 at 2.5 Mbit/s, and mux the EncodedVideoChunks back into an MP4 — a 60-second 4K clip drops from 337 MB to about 19 MB in ten seconds, and every frame you forget to close() is a hard leak.

WebCodecs is the only browser API that re-encodes video faster than real time, and it is deliberately incomplete: it gives you codecs and nothing else. No demuxer, no muxer, no scaler, no rate-control heuristics. This article sits in client-side media preprocessing under frontend UX, chunking and progress tracking, and unlike resizing images in the browser with canvas, the video equivalent is a genuine build, not a twenty-line helper.

When to use this approach #

• The upload is the bottleneck and the device has hardware to spare. A 337 MB clip on a 1.5 Mbit/s uplink takes thirty minutes and will not survive a lock screen. Ten seconds of the phone’s encode block turns it into a two-minute upload that never needs handling for 500 MB files.
• You need MP4 out, not WebM. MediaRecorder is forty lines instead of two hundred and fifty, but it runs at 1× playback speed and emits WebM in Chrome and Firefox. If your consumers want H.264 in an MP4 with a moov box at the front, WebCodecs plus a muxer is the browser’s only route.
• Not when quality, format breadth or battery matter more than bytes. One re-encode of a compressed source is a second generation of loss you cannot undo, and a phone at 8% battery will thermal-throttle its encoder. When any of that bites, ship the original and queue a transcode job with SQS and Lambda instead.

Prerequisites #

1. Chrome/Edge 94+, Safari 16.4+ or Firefox 130+ for VideoEncoder and VideoDecoder. Feature-detect with "VideoEncoder" in globalThis — there is no polyfill worth shipping.
2. A demuxer and a muxer. This page uses mp4box.js 0.5.x to read and mp4-muxer 5.x to write; both are dependency-free ESM and add roughly 60 KB gzipped together.
3. TypeScript 5.x with "lib": ["DOM", "DOM.Iterable", "ES2022", "WebWorker"] and @types/dom-webcodecs if your TS version predates the built-in definitions.
4. An MP4 or MOV source you have actually confirmed is one — file.type on an Android share-sheet pick is frequently empty, so verify with magic-byte detection before you spend a decode.

The shape of the pipeline #

Five stages, three of which you write yourself. The browser owns the decode and the encode; the container parsing, the resize and the container writing are yours.

Implementation #

One module. It demuxes in 8 MB slices, negotiates an encoder the device can accelerate, and runs a single-threaded loop with an explicit backpressure gate.

// transcode-video.ts
import MP4Box, { type MP4ArrayBuffer, type MP4File, type MP4Sample } from "mp4box";
import { ArrayBufferTarget, Muxer } from "mp4-muxer";

export interface TranscodeOptions {
  maxHeight: number;             // 1080 — cap on the output's short axis
  bitrate: number;               // 2_500_000 bits per second
  keyFrameIntervalSec: number;   // 2 — one IDR every two seconds
  onProgress?: (encoded: number, total: number) => void;
}

interface DemuxedTrack {
  samples: MP4Sample[];
  config: VideoDecoderConfig;
  width: number;
  height: number;
}

/** Pull avcC / hvcC / vpcC / av1C out of the sample entry, minus its 8-byte box header. */
function codecDescription(mp4: MP4File, trackId: number): Uint8Array {
  const trak = mp4.getTrackById(trackId);
  for (const entry of trak.mdia.minf.stbl.stsd.entries) {
    const box = entry.avcC ?? entry.hvcC ?? entry.vpcC ?? entry.av1C;
    if (!box) continue;
    const stream = new MP4Box.DataStream(undefined, 0, MP4Box.DataStream.BIG_ENDIAN);
    box.write(stream);
    return new Uint8Array(stream.buffer, 8);
  }
  throw new Error("video track carries no avcC/hvcC description box");
}

async function demux(file: File): Promise<DemuxedTrack> {
  const mp4 = MP4Box.createFile();
  const samples: MP4Sample[] = [];
  let video: any = null;
  let failure: Error | null = null;

  mp4.onError = (err: string) => { failure ??= new Error(`mp4box: ${err}`); };
  mp4.onReady = (info: any) => {
    video = info.videoTracks[0] ?? null;
    if (!video) return;
    mp4.setExtractionOptions(video.id, null, { nbSamples: 512 });
    mp4.start();
  };
  mp4.onSamples = (_id: number, _user: unknown, batch: MP4Sample[]) => { samples.push(...batch); };

  // appendBuffer wants a plain ArrayBuffer tagged with its byte offset in the file.
  const SLICE = 8 * 1024 * 1024;
  for (let offset = 0; offset < file.size; offset += SLICE) {
    const buf = (await file.slice(offset, offset + SLICE).arrayBuffer()) as MP4ArrayBuffer;
    buf.fileStart = offset;
    mp4.appendBuffer(buf);
    if (failure) throw failure;
  }
  mp4.flush();
  if (failure) throw failure;
  if (!video) throw new Error("no moov box parsed — not an MP4, or the file is truncated");

  return {
    samples,
    width: video.video.width,
    height: video.video.height,
    config: {
      codec: video.codec,                       // "avc1.640028", "hvc1.1.6.L150.B0", …
      codedWidth: video.video.width,
      codedHeight: video.video.height,
      description: codecDescription(mp4, video.id),
      hardwareAcceleration: "prefer-hardware",
    },
  };
}

async function pickEncoderConfig(
  width: number, height: number, framerate: number, bitrate: number,
): Promise<VideoEncoderConfig> {
  const base = { width, height, framerate, bitrate,
    bitrateMode: "variable", latencyMode: "quality" } as const;
  const candidates: VideoEncoderConfig[] = [
    // High 4.0: broadest hardware support, and every downstream player takes it.
    { ...base, codec: "avc1.640028", hardwareAcceleration: "prefer-hardware", avc: { format: "avc" } },
    // Main 4.0: some mobile encoder blocks refuse High profile above 1080p30.
    { ...base, codec: "avc1.4d0028", hardwareAcceleration: "prefer-hardware", avc: { format: "avc" } },
    // Let the browser choose software; expect roughly a quarter of the throughput.
    { ...base, codec: "avc1.640028", hardwareAcceleration: "no-preference", avc: { format: "avc" } },
  ];
  for (const candidate of candidates) {
    const support = await VideoEncoder.isConfigSupported(candidate);
    if (support.supported && support.config) return support.config;
  }
  throw new Error(`no H.264 encoder available for ${width}×${height}@${framerate}`);
}

/** Hold the feeder until both codec queues drain below `limit`. */
function gate(decoder: VideoDecoder, encoder: VideoEncoder, limit: number): Promise<void> {
  if (decoder.decodeQueueSize <= limit && encoder.encodeQueueSize <= limit) return Promise.resolve();
  return new Promise((resolve) => {
    const check = () => {
      if (decoder.decodeQueueSize > limit || encoder.encodeQueueSize > limit) return;
      decoder.removeEventListener("dequeue", check);
      encoder.removeEventListener("dequeue", check);
      clearInterval(poll);
      resolve();
    };
    // Not every engine fires `dequeue`, so keep a slow poll as the safety net.
    const poll = setInterval(check, 10);
    decoder.addEventListener("dequeue", check);
    encoder.addEventListener("dequeue", check);
  });
}

export async function transcode(file: File, opts: TranscodeOptions): Promise<Blob> {
  const track = await demux(file);

  // Hardware H.264 encoders reject odd dimensions — mask the low bit off both axes.
  const scale = Math.min(1, opts.maxHeight / track.height);
  const outW = Math.round(track.width * scale) & ~1;
  const outH = Math.round(track.height * scale) & ~1;

  const first = track.samples[0];
  const last = track.samples[track.samples.length - 1];
  const seconds = (last.cts + last.duration - first.cts) / first.timescale;
  const fps = Math.max(1, Math.round(track.samples.length / seconds));

  const decodable = await VideoDecoder.isConfigSupported(track.config);
  if (!decodable.supported) throw new Error(`this browser cannot decode ${track.config.codec}`);
  const encoderConfig = await pickEncoderConfig(outW, outH, fps, opts.bitrate);

  const muxer = new Muxer({
    target: new ArrayBufferTarget(),
    video: { codec: "avc", width: outW, height: outH },
    fastStart: "in-memory",        // moov before mdat, so the file plays while downloading
  });

  let fatal: Error | null = null;
  const onFatal = (e: DOMException) => { fatal ??= new Error(`${e.name}: ${e.message}`); };

  let encoded = 0;
  const encoder = new VideoEncoder({
    output: (chunk, meta) => {
      muxer.addVideoChunk(chunk, meta);
      opts.onProgress?.(++encoded, track.samples.length);
    },
    error: onFatal,
  });
  encoder.configure(encoderConfig);

  // WebCodecs has no scaler. The canvas is the scaler.
  const canvas = new OffscreenCanvas(outW, outH);
  const ctx = canvas.getContext("2d", { alpha: false, desynchronized: true })!;
  const keyEvery = Math.max(1, Math.round(fps * opts.keyFrameIntervalSec));
  let index = 0;

  const decoder = new VideoDecoder({
    output: (frame) => {
      try {
        ctx.drawImage(frame, 0, 0, outW, outH);
        const scaled = new VideoFrame(canvas, {
          timestamp: frame.timestamp,
          duration: frame.duration ?? 1e6 / fps,
        });
        encoder.encode(scaled, { keyFrame: index % keyEvery === 0 });
        scaled.close();
        index++;
      } finally {
        frame.close();     // non-negotiable — see the next section
      }
    },
    error: onFatal,
  });
  decoder.configure(track.config);

  for (const sample of track.samples) {
    if (fatal) throw fatal;
    await gate(decoder, encoder, 8);
    decoder.decode(new EncodedVideoChunk({
      type: sample.is_sync ? "key" : "delta",
      timestamp: (1e6 * sample.cts) / sample.timescale,
      duration: (1e6 * sample.duration) / sample.timescale,
      data: sample.data,
    }));
  }

  await decoder.flush();
  await encoder.flush();
  decoder.close();
  encoder.close();
  if (fatal) throw fatal;

  muxer.finalize();
  return new Blob([muxer.target.buffer], { type: "video/mp4" });
}

Line-by-line on the critical parameters #

• description: codecDescription(...). For avc1 and hvc1 the decoder needs the parameter sets from the avcC box, because MP4 samples are length-prefixed rather than Annex B start-code delimited. Omit it and configure() succeeds while the first decode() fails with EncodingError: Decoding error: Unexpected error. The new Uint8Array(stream.buffer, 8) skips the box’s own size and type fields — WebCodecs wants the payload only.
• avc: { format: "avc" } on the encoder. This asks for length-prefixed output plus an avcC blob in the chunk metadata, which is exactly what an MP4 muxer needs. The default is "annexb", which produces a file no player will open.
• hardwareAcceleration: "prefer-hardware". A preference, never a guarantee: isConfigSupported returns supported: true on a software fallback too. On an M2 the hardware path encodes 1080p at 380–450 fps; the software path manages 90–120. On a mid-range Android the gap is 110 fps versus 22, which is the difference between “faster than the upload” and “slower than real time”.
• bitrateMode: "variable" with latencyMode: "quality". Constant bitrate wastes bits on static shots and starves motion; quality latency mode lets the encoder use lookahead and B-frames. Use "realtime" only for a live stream, where it disables lookahead to keep per-frame latency under a frame interval.
• & ~1 on the output dimensions. H.264 chroma is subsampled 2:1, so odd dimensions are rejected outright by most hardware encoders — usually as NotSupportedError at configure() time, occasionally as a green column down the right edge.
• { keyFrame: index % keyEvery === 0 }. There is no GOP-length field in VideoEncoderConfig. If you never ask for a keyframe you get exactly one, at frame zero, and seeking in the result means decoding from the start.
• nbSamples: 512. mp4box batches onSamples callbacks. Larger batches mean fewer callbacks and more retained sample data; 512 keeps the retained set around 40 MB for 4K.

Backpressure and the frames you must close #

A hardware decoder is typically two to four times faster than the encoder on the same chip. Feed it without restraint and the decoder happily produces frames the encoder cannot consume, and each of those frames is an uncompressed picture: 1920×1080 NV12 is 3.1 MB, 3840×2160 is 12.4 MB. A hundred queued 4K frames is 1.2 GB of non-JS-heap memory, and the renderer is killed without an exception you can catch.

The gate above waits on the dequeue event and falls back to a 10 ms poll. A limit of 8 is a good default: deep enough that the encoder never idles, shallow enough that peak frame memory stays under 100 MB even at 4K. Raising it to 64 buys no throughput and multiplies the memory.

Frame closure is separate and stricter. VideoFrame holds a platform surface that no GC can free on your behalf, so the API requires an explicit close(). Miss it and Chrome logs A VideoFrame was garbage collected without being closed. Applications should call close() on frames when done with them to prevent stalls. — and then the decoder, having run out of its fixed pool of output buffers, simply stops emitting. No error, no rejection, just a flush() that never resolves. Both the source frame and the canvas-derived frame need closing; the try/finally above is the shape that survives an encoder throwing mid-loop.

Choosing bitrate and keyframe cadence #

Bitrate is the whole compression ratio, and resolution is most of the perceived quality. These are the settings worth shipping for user-generated clips:

Keyframes cost five to ten times a delta frame. At keyFrameIntervalSec: 2 and 30 fps, roughly 8% of the output is keyframe data and a player can seek to any two-second boundary. Push it to 10 seconds and you save about 5% of the bytes while making scrubbing feel broken; drop it to 1 second and you pay 15% for granularity nobody asked for. Two seconds is also what HLS and DASH packagers expect, so a clip encoded this way segments cleanly later without a second pass through an FFmpeg job on the server.

Record the output’s dimensions, duration and bitrate with the upload rather than re-probing later; there is a schema for exactly this in storing image dimensions and duration metadata.

Pulling frames without a demuxer #

If parsing MP4 boxes is more than you want to own, Chromium lets you play the file into a MediaStreamTrackProcessor and read VideoFrames straight off a stream:

const video = document.createElement("video");
video.src = URL.createObjectURL(file);
video.muted = true;
await video.play();

const stream = (video as HTMLVideoElement & { captureStream(): MediaStream }).captureStream();
const reader = new MediaStreamTrackProcessor({ track: stream.getVideoTracks()[0] }).readable.getReader();

for (;;) {
  const { value: frame, done } = await reader.read();
  if (done) break;
  encoder.encode(frame);
  frame.close();
}
await encoder.flush();

Twelve lines instead of eighty, and three real costs. It runs at playback speed, so a 60-second clip takes 60 seconds. It drops frames whenever the compositor is busy, so the output is not frame-accurate. And MediaStreamTrackProcessor is Chromium-only — Safari throws ReferenceError: Can't find variable: MediaStreamTrackProcessor. Use it for a proof of concept or for a genuinely live source; use the demuxer for files.

Configuration gotchas #

NotSupportedError: Unsupported configuration. Check isConfigSupported() prior to calling configure(). Thrown by configure() for odd dimensions, for a resolution above the encoder block’s ceiling (many mobile H.264 encoders stop at 1920×1088), or for a codec string whose level is too low for the frame size you asked for — avc1.42001f is Level 3.1 and caps at 1280×720. Always run the candidate ladder and use the returned support.config, which the browser has normalised for you.

EncodingError: Encoding error: Encoder failure. The hardware encoder session could not be acquired or died mid-stream. Most devices allow only two to four concurrent sessions, so a second tab running the same code will trip this. Catch it in the error callback, reconfigure with hardwareAcceleration: "prefer-software" and restart from the last keyframe, or abandon the transcode and upload the original.

InvalidStateError: Failed to execute 'encode' on 'VideoEncoder': Cannot call 'encode' on a closed codec. A late decoder output arrived after you closed the encoder. It happens when you await encoder.flush() before await decoder.flush() — the decoder still has frames in flight that will call encoder.encode(). Flush the decoder first, always.

RangeError: Invalid array length from mp4box.js. The moov box sits at the end of the file, which is normal for clips recorded on Android and for anything written by a naive encoder. onReady cannot fire until you have appended every byte, so a 337 MB file is fully resident before demuxing starts. Slice it in with Blob.slice as the code above does — slicing large files with Blob.slice covers why reading the whole thing with arrayBuffer() is the version that fails on a phone.

When a server transcode is the right answer #

Four situations where the server is simply correct. You need multiple renditions: producing 240p, 480p and 1080p ladders on a phone means three encode passes and three uploads. The source is not MP4 or WebM: AVI, MKV, MXF, ProRes and anything from a drone or an action camera will not demux with a 60 KB library, and VideoDecoder may not have the codec at all. Correctness is contractual: broadcast delivery, colour-managed workflows and HDR-to-SDR tone mapping are not things a browser encoder does well or predictably. Your users are on Safari and the transcode is mandatory: WebCodecs works there from 16.4, but iOS 16 devices are still in the field, and a feature that silently does nothing for a slice of your users is worse than one that never existed.

The pragmatic pattern is both. Transcode on the client when VideoEncoder.isConfigSupported says yes and the file is over some threshold, upload the original otherwise, and let the server normalise everything after the fact. Whichever path runs, surface a distinct “Compressing…” phase in the UI — a ten-second silence after a tap reads as a broken button, and the time-remaining estimate you show during the upload should not start until the encode has finished.

Verification #

Transcode a known clip and assert the four properties that matter: it is smaller, the dimensions are what you asked for, the duration is unchanged, and the realised bitrate is near the target.

import { transcode } from "./transcode-video.js";

const file = (document.querySelector("input[type=file]") as HTMLInputElement).files![0];
const t0 = performance.now();
const out = await transcode(file, {
  maxHeight: 1080,
  bitrate: 2_500_000,
  keyFrameIntervalSec: 2,
  onProgress: (done, total) => { if (done % 60 === 0) console.log(`${done}/${total}`); },
});
const seconds = (performance.now() - t0) / 1000;

const probe = document.createElement("video");
probe.src = URL.createObjectURL(out);
await new Promise((r) => probe.addEventListener("loadedmetadata", r, { once: true }));

console.assert(out.size < file.size / 4, `only ${(file.size / out.size).toFixed(1)}× smaller`);
console.assert(probe.videoHeight === 1080, `height is ${probe.videoHeight}`);
console.log(
  `${probe.videoWidth}×${probe.videoHeight} · ${probe.duration.toFixed(2)} s · ` +
  `${(out.size / 1e6).toFixed(1)} MB · ${((out.size * 8) / probe.duration / 1e6).toFixed(2)} Mbit/s · ` +
  `encoded in ${seconds.toFixed(1)} s`,
);
URL.revokeObjectURL(probe.src);

Expected for a 60-second 4K/30 iPhone clip on an M2 MacBook:

1920×1080 · 60.03 s · 18.9 MB · 2.52 Mbit/s · encoded in 9.4 s

A realised bitrate more than 20% above target usually means the keyframe interval is too short. A duration that is short by a second or two means frames were dropped — check that you closed every VideoFrame and that the loop awaited gate() before every decode(). Then upload the blob exactly as you would the original, with the same resumable upload state machine around it, because 19 MB on a train still fails.

Frequently Asked Questions