Generating Video Thumbnails with FFmpeg in Node.js #

Probe the container for its duration first, then spawn ffmpeg twice — once with -ss before -i and -frames:v 1 for the poster, once with fps plus tile for the scrubbing sprite — and pipe both JPEGs out of stdout into S3 without ever touching the local disk.

This article sits under post-upload media transcoding within backend validation and cloud storage architecture. It assumes the upload has already completed and passed whatever content checks you run — the derived assets it produces are what you record via storing image dimensions and duration metadata.

When to use this approach #

• You control the runtime and can ship an ffmpeg binary with it — a Lambda layer, a container image, or a Fargate task. Two spawns on a 4-minute 1080p H.264 file take roughly 1.5 s and 6 s respectively on two vCPUs, which fits inside almost any job timeout.
• You want the exact frames you asked for. A managed service such as MediaConvert produces thumbnails as a side effect of a transcode job, bills per output minute, and takes tens of seconds to queue; running the binary yourself costs one process and returns bytes.
• You are producing images, not video. Piping an MP4 out of stdout is a different problem, because the moov atom is written last and a pipe cannot seek backwards. JPEG and PNG have no such constraint.

Reach for a wrapper library only if you need its filter-graph builder. fluent-ffmpeg is unmaintained, hides the argument array you eventually need to read in a log line, and its .screenshots() helper writes temp files. Everything below uses node:child_process directly.

Prerequisites #

1. Node 20.11+ and a static ffmpeg 6.1 or 7.x build with --enable-libx264 and the mjpeg encoder. Check with ffmpeg -hide_banner -encoders | grep mjpeg.
2. @aws-sdk/client-s3 and @aws-sdk/lib-storage, both 3.600.0 or later. lib-storage is what accepts a Readable as a body.
3. FFMPEG_PATH and FFPROBE_PATH in the environment. Never resolve the binary through PATH in production — you want the failure to be a config error at boot, not an ENOENT at 3 a.m.
4. An IAM role with s3:GetObject on the source prefix and s3:PutObject plus s3:AbortMultipartUpload on the derivatives prefix.
5. A source object your process can read. Either fetch it to a stream or hand ffmpeg an HTTPS URL produced by generating secure presigned URLs with AWS SDK v3; its HTTP protocol handler issues Range requests and will seek rather than download the whole file.

The shape of the job #

Nothing else can be decided until you know the duration. A poster at “10% in” needs it, the sprite interval is derived from it, and a seek past the end of the file produces a zero-byte object that no exit code will warn you about. So ffprobe runs first, once, and its JSON feeds both spawns.

Implementation #

One helper spawns a bounded process; the other two use it. This is the whole poster path.

import { spawn } from "node:child_process";
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";

const FFMPEG = process.env.FFMPEG_PATH ?? "/opt/bin/ffmpeg";
const FFPROBE = process.env.FFPROBE_PATH ?? "/opt/bin/ffprobe";
const s3 = new S3Client({});

export class FFmpegError extends Error {
  constructor(
    readonly bin: string,
    readonly code: number | null,
    readonly signal: NodeJS.Signals | null,
    readonly stderr: string,
    readonly timedOut: boolean,
  ) {
    super(
      `${bin} exited code=${code} signal=${signal} timedOut=${timedOut}\n` +
        stderr.slice(-1200),
    );
    this.name = "FFmpegError";
  }
}

/** Spawn a binary, keep the last 8 KB of stderr, and guarantee it dies. */
function spawnBounded(bin: string, args: string[], timeoutMs: number) {
  const child = spawn(bin, args, { stdio: ["ignore", "pipe", "pipe"] });

  let stderr = "";
  child.stderr.setEncoding("utf8");
  child.stderr.on("data", (c: string) => {
    stderr = (stderr + c).slice(-8192); // ring buffer — a bad file is chatty
  });

  let timedOut = false;
  const timer = setTimeout(() => {
    timedOut = true;
    child.kill("SIGTERM");
    // A process blocked in a stalled TLS read never reaps its own SIGTERM.
    setTimeout(() => child.kill("SIGKILL"), 5_000).unref();
  }, timeoutMs);

  const exited = new Promise<void>((resolve, reject) => {
    child.on("error", reject); // ENOENT: the binary is not where you think
    child.on("close", (code, signal) => {
      clearTimeout(timer);
      if (code === 0) resolve();
      else reject(new FFmpegError(bin, code, signal, stderr, timedOut));
    });
  });

  return { child, exited };
}

export interface Probe {
  durationSec: number;
  width: number;
  height: number;
}

export async function probeVideo(input: string): Promise<Probe> {
  const { child, exited } = spawnBounded(
    FFPROBE,
    ["-v", "error", "-select_streams", "v:0", "-print_format", "json",
     "-show_format", "-show_streams", input],
    15_000,
  );

  let json = "";
  child.stdout.setEncoding("utf8");
  child.stdout.on("data", (c: string) => { json += c; });
  await exited;

  const out = JSON.parse(json);
  const v = out.streams?.[0];
  if (!v) throw new Error("no video stream — audio-only or unreadable input");

  const durationSec = Number(out.format?.duration ?? v.duration ?? NaN);
  if (!Number.isFinite(durationSec) || durationSec <= 0) {
    throw new Error("container reports no duration; refuse to guess a seek");
  }

  // A phone recording carries a display matrix; width/height are pre-rotation.
  const rot = Math.abs(
    Number(v.side_data_list?.find(
      (s: { rotation?: number }) => s.rotation !== undefined)?.rotation ?? 0),
  ) % 180;
  const swap = rot === 90;

  return {
    durationSec,
    width: swap ? Number(v.height) : Number(v.width),
    height: swap ? Number(v.width) : Number(v.height),
  };
}

export async function posterFrame(o: {
  input: string; bucket: string; key: string; atSec: number;
  boxW?: number; boxH?: number;
}) {
  const boxW = o.boxW ?? 1280;
  const boxH = o.boxH ?? 720;

  const args = [
    "-hide_banner", "-loglevel", "error",
    "-ss", o.atSec.toFixed(3),      // BEFORE -i: the demuxer jumps, no decode
    "-i", o.input,
    "-an", "-sn", "-dn",            // drop audio, subtitle and data streams
    "-frames:v", "1",
    "-vf", [
      "thumbnail=n=40",             // score 40 frames, emit the least average
      `scale=w=${boxW}:h=${boxH}:force_original_aspect_ratio=decrease:flags=lanczos`,
      `pad=${boxW}:${boxH}:(ow-iw)/2:(oh-ih)/2:color=black`,
      "format=yuvj420p",
    ].join(","),
    "-f", "image2",                 // pipe:1 has no extension to infer from
    "-c:v", "mjpeg", "-q:v", "3",   // 2–5 is the useful band; 3 ≈ 85 KB at 720p
    "-update", "1",
    "pipe:1",
  ];

  const { child, exited } = spawnBounded(FFMPEG, args, 20_000);
  child.stdout.on("error", () => {}); // EPIPE if the upload aborts first

  const upload = new Upload({
    client: s3,
    params: {
      Bucket: o.bucket, Key: o.key, Body: child.stdout,
      ContentType: "image/jpeg",
      CacheControl: "public, max-age=31536000, immutable",
    },
    queueSize: 1,          // stdout is one sequential stream; no parallelism
    partSize: 5 * 1024 * 1024,
  });

  const [exit, put] = await Promise.allSettled([exited, upload.done()]);
  if (exit.status === "rejected") {
    await upload.abort().catch(() => {});
    throw exit.reason;
  }
  if (put.status === "rejected") throw put.reason;
  return put.value;
}

The parameters that matter #

• -ss before -i is input seeking: the demuxer seeks the file, then decodes forward from the preceding keyframe. Putting it after -i is output seeking, which decodes and discards everything from zero. On a 40-minute file that is the difference between 0.4 s and 90 s. Since ffmpeg 2.1 input seeking is frame-accurate anyway, so there is no correctness argument left for the slow form.
• -frames:v 1 stops the encoder after one frame. Without it, -ss alone happily encodes to the end of the file.
• thumbnail=n=40 buffers 40 frames, compares each one’s RGB histogram against the batch average, and forwards the most atypical. That is roughly 1.6 s of decode at 25 fps and it is what saves you from black frames and fades.
• force_original_aspect_ratio=decrease fits the frame inside the 1280×720 box without distortion, leaving one dimension short. The following pad centres it and fills the gap, so every poster in your grid is exactly 1280×720 regardless of whether the source was 4:3, 9:16 or 2.39:1. Swap decrease for increase plus crop=1280:720 if you would rather lose edges than show bars.
• format=yuvj420p pins the full-range JPEG pixel format. Without it a 10-bit or 4:2:2 source makes the mjpeg encoder pick yuvj422p, which some older image decoders reject.
• -update 1 tells the image2 muxer that repeated writes overwrite one output rather than needing a %03d pattern. It is what makes pipe:1 legal.
• -an -sn -dn matter more than they look. A file with a cover-art stream will otherwise have that stream selected as the “best video stream” and your poster becomes the album art.
• The stderr ring buffer keeps the last 8 KB. Log the full thing on failure and only the tail in the exception message; a corrupt H.264 file can emit tens of megabytes of Invalid NAL unit size before it gives up.

Choosing a frame that is not black #

The single most common bug in a thumbnail pipeline is a wall of black rectangles. Videos open on fades, slates and letterboxed title cards, so -ss 0 is almost always wrong. A fixed offset is better but still guesses. Combining a proportional seek with the thumbnail filter is what actually holds up across a mixed corpus: seek to 10% of the duration, then let the filter pick the most visually distinct frame from the next 40.

Two caveats keep this honest. The filter’s window is bounded by n, so if the first 40 frames after your seek are all black you get a black frame back; clamp the seek with Math.min(duration * 0.1, 30) and consider a second attempt at 50% when the first result’s mean luma is under 16. And thumbnail forces a decode of n frames, which costs about 200 ms at 1080p — negligible next to the seek you already saved.

The sprite sheet for scrubbing previews #

A scrub preview needs a few hundred small frames in one request, not a few hundred requests. One ffmpeg invocation with fps → scale → pad → tile produces the entire grid, and the interval falls straight out of the duration you probed.

export function spriteArgs(input: string, durationSec: number, opts = {
  cols: 10, rows: 10, tileW: 160, tileH: 90,
}) {
  const tiles = opts.cols * opts.rows;
  // Nudge the interval down so rounding cannot leave the last cell grey.
  const interval = Math.max((durationSec / tiles) * 0.999, 0.25);

  return {
    interval,
    args: [
      "-hide_banner", "-loglevel", "error",
      "-skip_frame", "nokey",        // decode keyframes only: 5–20× faster
      "-i", input,
      "-an", "-sn", "-dn",
      "-vf", [
        `fps=1/${interval.toFixed(4)}`,
        `scale=w=${opts.tileW}:h=${opts.tileH}:force_original_aspect_ratio=decrease`,
        `pad=${opts.tileW}:${opts.tileH}:(ow-iw)/2:(oh-ih)/2:color=black`,
        `tile=${opts.cols}x${opts.rows}:margin=0:padding=0:color=black`,
      ].join(","),
      "-frames:v", "1",              // one composed sheet, not one per tile
      "-f", "image2", "-c:v", "mjpeg", "-q:v", "6", "-update", "1",
      "pipe:1",
    ],
  };
}

The player needs a WebVTT track to map time to pixels, and it is pure arithmetic over the same two numbers:

const pad = (s: number) =>
  new Date(s * 1000).toISOString().slice(11, 23); // HH:MM:SS.mmm

export function spriteVtt(url: string, interval: number, count: number,
                          cols: number, tileW: number, tileH: number) {
  const cues = Array.from({ length: count }, (_, i) => {
    const x = (i % cols) * tileW;
    const y = Math.floor(i / cols) * tileH;
    return `${pad(i * interval)} --> ${pad((i + 1) * interval)}\n` +
           `${url}#xywh=${x},${y},${tileW},${tileH}`;
  });
  return `WEBVTT\n\n${cues.join("\n\n")}\n`;
}

Two production notes. -skip_frame nokey makes the decode enormously cheaper by ignoring non-keyframes, but the fps filter then duplicates the nearest keyframe to fill each slot, so tiles land within one GOP — usually 2 s — of their nominal time. That is invisible in a scrub strip and worth the speedup on anything over ten minutes. And keep the sheet under about 4000 px in either direction: Safari on iOS silently downsamples larger canvases, which turns a crisp strip into mush. Past roughly 200 tiles, emit several sheets and point later cues at the second URL.

Streaming stdout to S3 instead of writing temp files #

Writing /tmp/poster.jpg and uploading it afterwards costs you a Lambda’s 512 MB ephemeral disk, a cleanup path in every error branch, and one full serialisation before a single byte moves. Passing child.stdout to lib-storage’s Upload removes all three: the muxer’s writes become part buffers, and a 120 KB poster goes up as a single PutObject because it never reaches the 5 MB part threshold. The same reasoning drives streaming file uploads in Node.js with Web Streams on the ingest side.

The rule that decides whether this is legal is whether the output muxer needs to seek. JPEG, PNG and WebP write forward only and pipe cleanly. MP4 writes its moov index last and then rewinds to patch offsets, which a pipe cannot do — you would need -movflags +frag_keyframe+empty_moov, producing a fragmented file that some players dislike. If you are also generating clips, write those to disk.

Three things to hold onto. Attach an error listener to child.stdout or a consumer that goes away throws an unhandled EPIPE and takes the worker down. Set queueSize: 1, because a single sequential pipe cannot feed parallel part uploads and a higher value just buys buffering. And never write the database row from the upload’s promise — write it after the exit code is zero, because a failed ffmpeg still closes stdout cleanly and S3 will happily store the truncated result.

Configuration gotchas #

The input is truncated #

An object created by an aborted multipart upload, or copied while still being written, produces this on stderr:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55d3f0a1c880] stream 0, offset 0x1a34c1: partial file
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55d3f0a1c880] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
pipe:0: Invalid data found when processing input

partial file means the mdat payload is shorter than the index claims. Its sibling, moov atom not found, means the index never arrived at all — typical when a browser upload was cancelled and the file was not written with -movflags +faststart. Both exit 1 with Invalid data found when processing input (AVERROR_INVALIDDATA, -1094995529). The fix is upstream: trigger the job from CompleteMultipartUpload rather than any earlier event, and expire the debris using S3 lifecycle rules for temporary uploads. If a real user file is genuinely damaged, -err_detect ignore_err will often still yield a poster from the readable prefix.

Unable to find a suitable output format for 'pipe:1' #

[out#0 @ 0x5583f6a3c400] Unable to find a suitable output format for 'pipe:1'
pipe:1: Invalid argument

pipe:1 has no extension, so the muxer cannot be guessed — pass -f image2 explicitly. Its close relative appears when you supply -f image2 but forget -update 1 and the graph emits more than one frame:

[image2 @ 0x55a1b4409c00] Could not get frame filename number 2 from pattern 'pipe:1'. Use '-frames:v 1' for a single image, or '-update' option, or use a pattern such as %03d within the filename.
av_interleaved_write_frame(): Invalid argument

A seek past the end produces a zero-byte object #

Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used); this may be not what you want.

Historically ffmpeg has returned exit status 0 alongside this message, so an exit-code check will not catch it and you will store a valid, empty S3 object. Clamp the seek to duration - 1, and assert on the uploaded byte count — anything under 1024 bytes is not a JPEG. This is the same class of bug as trusting a Content-Type header instead of the bytes, covered in detecting file type from magic bytes in JavaScript.

The process outlives its timeout #

child.kill("SIGTERM") is a request. An ffmpeg blocked in a stalled HTTPS read — an S3 connection that neither delivers nor resets — will not act on it, and in a container the orphan holds a CPU until the task is reaped. Always follow with SIGKILL on a short unref()ed timer, as the helper above does, and set -rw_timeout 15000000 (microseconds) on HTTP inputs so the read gives up on its own. Budget generously: 20 s for a poster, 120 s for a sprite over an hour-long file.

Verification #

Round-trip the derivative through ffprobe rather than trusting the exit code:

#!/usr/bin/env bash
set -euo pipefail
BUCKET=acme-media
KEY=derived/f1a9/poster.jpg

BYTES=$(aws s3api head-object --bucket "$BUCKET" --key "$KEY" \
  --query ContentLength --output text)
[ "$BYTES" -gt 1024 ] || { echo "FAIL: $BYTES-byte poster"; exit 1; }

aws s3 cp "s3://$BUCKET/$KEY" - | ffprobe -v error -print_format json \
  -show_entries stream=codec_name,width,height -i pipe:0

Expected output, exactly:

{
    "programs": [],
    "streams": [
        {
            "codec_name": "mjpeg",
            "width": 1280,
            "height": 720
        }
    ]
}

Then guard against the black-frame regression in CI. ffmpeg -i poster.jpg -vf "signalstats,metadata=print:key=lavfi.signalstats.YAVG" -f null - prints one YAVG line; fail the build under 16 or over 240. Run it against a fixture set that includes a fade-in, a portrait phone clip and a 3-second file. If your workers run as Lambda, the same fixtures belong in the harness described in serverless virus scanning with AWS Lambda, since both jobs share the ObjectCreated trigger.

Frequently Asked Questions