Queueing Transcode Jobs with SQS and Lambda #

Point the bucket’s ObjectCreated notification at a standard SQS queue, attach a Lambda event source mapping with BatchSize: 5 and FunctionResponseTypes: ["ReportBatchItemFailures"], and set the queue’s VisibilityTimeout to six times the function timeout — almost every production problem in this pipeline is a consequence of getting one of those three numbers wrong.

This article sits under post-upload media transcoding within backend validation and cloud storage architecture. It covers only the plumbing between the upload landing in a bucket and a worker picking it up; what the worker does with the bytes is the subject of building an image derivative pipeline with Sharp and generating video thumbnails with FFmpeg in Node.js.

When to use this approach #

• Your arrival rate is spiky and your processing rate is not. A marketing team drops 4,000 photos in ninety seconds; a queue turns that into a backlog you drain at a chosen concurrency instead of 4,000 simultaneous invocations fighting over a database connection pool.
• The work can fail transiently and must be retried without a human. S3 can be throttled, a decoder can OOM, a downstream API can 503. SQS gives you redelivery with an increasing ApproximateReceiveCount and a dead-letter queue as the terminal state.
• Each job finishes inside the Lambda ceiling — 15 minutes, and realistically under 10 so a retry still fits in the visibility window. Above that, the queue should carry a submission to AWS Elemental MediaConvert or a Fargate task, with completion arriving as a second event.

Skip the queue only when the work is trivial and idempotent — writing a metadata row, tagging an object. S3 can invoke Lambda directly, but that path has no batching, no message-level retry policy you control, and asynchronous invocation gives you exactly two retries before the event vanishes.

Prerequisites #

1. Node.js 20 (nodejs20.x or later) and @aws-sdk/client-sqs 3.600+. The AWS SDK is available in the runtime, but pin it in package.json anyway — the bundled version changes under you.
2. Two queues: transcode-jobs and transcode-jobs-dlq, in the same region and account as the bucket.
3. A queue policy that lets S3 write to it. Without the aws:SourceArn condition anyone in your account can enqueue jobs; without the policy at all, the bucket notification refuses to save.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowBucketNotifications",
    "Effect": "Allow",
    "Principal": { "Service": "s3.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:eu-west-1:111122223333:transcode-jobs",
    "Condition": {
      "StringEquals": { "aws:SourceAccount": "111122223333" },
      "ArnLike": { "aws:SourceArn": "arn:aws:s3:::acme-uploads" }
    }
  }]
}

4. An execution role with sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes and sqs:ChangeMessageVisibility on the job queue. The first three are what the event source mapping uses on your behalf; the fourth is for the heartbeat below.

Why the queue is not optional #

The bucket cannot apply backpressure. It accepts every PutObject and every CompleteMultipartUpload at whatever rate the internet offers, and it will happily fan a burst straight into your compute. The queue is the shock absorber: messages accumulate, ApproximateNumberOfMessages climbs, and the event source mapping adds pollers at a bounded rate — 60 more concurrent batches per minute, up to 1,250 — so your database and your image encoders see a ramp rather than a wall.

Two properties are worth naming because they drive every design decision below. S3 event delivery is at-least-once — a single upload can produce two notifications, and the docs make no promise otherwise. And a standard queue is also at-least-once, so the duplicate is preserved end to end. Idempotency is therefore a requirement of the worker, not an optimisation, and the cheapest key is the object’s ETag rather than its name; the same argument as retrying fetch uploads with idempotency keys makes on the client side.

Implementation #

One handler. It parses the S3 envelope out of the SQS body, keeps the message invisible while a long job runs, and returns the identifiers of exactly the messages that failed.

import { SQSClient, ChangeMessageVisibilityCommand } from "@aws-sdk/client-sqs";
import type {
  Context, SQSBatchItemFailure, SQSBatchResponse, SQSEvent, SQSRecord,
} from "aws-lambda";
import { transcode, UnprocessableInput } from "./transcode.js";

const sqs = new SQSClient({});
const QUEUE_URL = process.env.JOB_QUEUE_URL!;
const HEARTBEAT_MS = 60_000;
const EXTEND_TO_S = 900;

interface S3Notification {
  Records?: Array<{
    eventName: string;
    s3: {
      bucket: { name: string };
      object: { key: string; size: number; eTag: string; versionId?: string };
    };
  }>;
  Event?: string; // s3:TestEvent, sent once when you save the notification
}

/** Push the invisibility deadline forward while one long job runs. */
function heartbeat(record: SQSRecord, ctx: Context): () => void {
  const timer = setInterval(() => {
    // Do not extend past our own death — let the message come back sooner.
    if (ctx.getRemainingTimeInMillis() < HEARTBEAT_MS) return;
    sqs
      .send(new ChangeMessageVisibilityCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: record.receiptHandle,
        VisibilityTimeout: EXTEND_TO_S,
      }))
      .catch((err) => console.warn("heartbeat failed", record.messageId, String(err)));
  }, HEARTBEAT_MS);
  timer.unref();
  return () => clearInterval(timer);
}

export async function handler(
  event: SQSEvent,
  ctx: Context,
): Promise<SQSBatchResponse> {
  const batchItemFailures: SQSBatchItemFailure[] = [];
  const poisonedGroups = new Set<string>();

  for (const record of event.Records) {
    const group = record.attributes.MessageGroupId; // undefined on a standard queue
    if (group !== undefined && poisonedGroups.has(group)) {
      // FIFO ordering: nothing behind a failure in the same group may run.
      batchItemFailures.push({ itemIdentifier: record.messageId });
      continue;
    }

    const stopHeartbeat = heartbeat(record, ctx);
    try {
      const body = JSON.parse(record.body) as S3Notification;
      if (body.Event === "s3:TestEvent") continue; // console "Save" probe, ack it
      const r = body.Records?.[0];
      if (!r) throw new Error(`unrecognised envelope: ${record.body.slice(0, 200)}`);

      // S3 URL-encodes the key and writes spaces as '+'.
      const key = decodeURIComponent(r.s3.object.key.replace(/\+/g, " "));

      await transcode({
        bucket: r.s3.bucket.name,
        key,
        etag: r.s3.object.eTag,   // idempotency key: the bytes, not the name
        versionId: r.s3.object.versionId,
        attempt: Number(record.attributes.ApproximateReceiveCount),
      });
    } catch (err) {
      if (err instanceof UnprocessableInput) {
        // A 40,000-pixel PNG will never decode. Five retries buy five failures.
        console.error(JSON.stringify({
          msg: "unprocessable", id: record.messageId, reason: err.message,
        }));
        continue; // acknowledged: the message is deleted, not redelivered
      }
      console.error(JSON.stringify({
        msg: "retryable",
        id: record.messageId,
        attempt: record.attributes.ApproximateReceiveCount,
        err: String(err),
      }));
      batchItemFailures.push({ itemIdentifier: record.messageId });
      if (group !== undefined) poisonedGroups.add(group);
    } finally {
      stopHeartbeat();
    }
  }

  return { batchItemFailures };
}

And the wiring, which is where the numbers live:

aws lambda create-event-source-mapping \
  --function-name transcode-worker \
  --event-source-arn arn:aws:sqs:eu-west-1:111122223333:transcode-jobs \
  --batch-size 5 \
  --maximum-batching-window-in-seconds 5 \
  --function-response-types ReportBatchItemFailures \
  --scaling-config MaximumConcurrency=40

The parameters that matter #

• FunctionResponseTypes: ["ReportBatchItemFailures"] is what makes the return value mean anything. Omit it and Lambda ignores batchItemFailures entirely: your handler returns a tidy list of one failure, Lambda deletes nothing, and all five messages come back. There is no warning and no metric for this.
• --batch-size 5 with a 5-second batching window. Larger batches amortise the invocation overhead, but every message in a batch shares one function timeout — five 90-second transcodes need 450 seconds of budget. Size the batch so batchSize × p99Job < functionTimeout × 0.8.
• MaximumConcurrency=40 caps how many batches are in flight. Use this rather than reserved concurrency on the function: a throttled invocation still counts as a receive, so reserved concurrency quietly drives healthy messages towards the dead-letter queue. The valid range is 2 to 1,000.
• record.attributes.ApproximateReceiveCount is the honest retry counter. Log it on every failure; a job failing at attempt 1 and a job failing at attempt 4 are different incidents, and the second one is about to become a dead-letter alarm.
• continue versus pushing to batchItemFailures is the single most consequential branch in the handler. Falling through to continue acknowledges the message — Lambda deletes it. Reserve that for input that is permanently unprocessable, and write the reason somewhere durable first, along the lines of the quarantine bucket patterns for infected uploads.
• timer.unref() stops the heartbeat interval from holding the event loop open. Without it, a Lambda that finishes its work still waits for the next tick before freezing the sandbox.

Visibility timeout arithmetic #

This is the failure you will hit. SQS’s default visibility timeout is 30 seconds. A 1080p poster-frame extraction plus three renditions takes 90. At t=30 the message becomes visible again while your first invocation is still decoding, a poller hands it to a second invocation, and now two functions are writing the same derivative key with different in-flight buffers.

The rule AWS documents, and the one to actually use, is VisibilityTimeout ≥ 6 × functionTimeout + MaximumBatchingWindowInSeconds. The factor of six is not superstition: a throttled or errored batch is retried by the mapping itself, and the extra headroom lets those retries happen inside a single visibility period instead of releasing the message to a different poller mid-retry. With a 900-second function that is 5,400 seconds, comfortably under the 12-hour ceiling.

When job length varies by two orders of magnitude — a 200 KB avatar and a 4 GB ProRes master on the same queue — do not size the timeout for the worst case, because that is also how long a genuinely stuck message stays invisible. Use a moderate timeout plus the ChangeMessageVisibility heartbeat in the handler above. The extension is measured from now, not from first receipt, and the total cannot exceed 12 hours from the original receive.

Partial batch responses #

Without ReportBatchItemFailures, a batch is atomic from Lambda’s point of view: throw once and all five messages return to the queue. Four of them already succeeded, so their side effects happen again, their ApproximateReceiveCount climbs on every round, and after five rounds four perfectly healthy jobs land in the dead-letter queue alongside the one poison message.

The response shape is unforgiving, and every one of these mistakes makes Lambda replay the entire batch: a key other than batchItemFailures, an itemIdentifier that is null, empty, or does not match a messageId in the event, or a non-JSON return. Returning undefined is treated as total success — which is why a handler that swallows errors silently loses jobs. Build the array explicitly, as above, and unit-test the object it returns.

Serialising an asset with FIFO and MessageGroupId #

Standard queues do not order anything. If your upload path emits “generate derivatives” and then “publish”, the publish message can be delivered first. Two fixes exist: make the worker order-insensitive, or use a FIFO queue with MessageGroupId set to the asset id. Messages in one group are delivered strictly in order and the next one is not released until the previous is deleted; different groups run fully in parallel, so a per-asset group id costs you nothing in throughput.

The catch is that S3 event notifications cannot target a FIFO queue. A FIFO queue needs MessageGroupId on every send and the notification service does not supply one, so the bucket configuration is rejected. If you need ordering, the producer must be your own code — the handler that completes the upload sends the message itself:

import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({});

export async function enqueueTranscode(assetId: string, key: string, etag: string) {
  await sqs.send(new SendMessageCommand({
    QueueUrl: process.env.FIFO_QUEUE_URL!,
    MessageBody: JSON.stringify({ assetId, key, etag }),
    MessageGroupId: assetId,          // ordering scope: one asset
    MessageDeduplicationId: `${etag}:derivatives`, // 5-minute dedup window
  }));
}

Three FIFO constraints to plan around. Batch size is capped at 10. Concurrency is bounded by the number of active groups, so a backlog of 500 assets scales to 500 in-flight batches but a backlog of 500 messages for one asset runs strictly serially. And a plain FIFO queue is limited to 300 API calls per second per action, or 3,000 messages a second with 10-message batches; past that, enable high-throughput mode with FifoThroughputLimit=perMessageGroupId and DeduplicationScope=messageGroup.

Configuration reference #

Configuration gotchas #

The visibility timeout check runs once, at creation #

Create a mapping against a queue whose timeout is too short and the API refuses:

An error occurred (InvalidParameterValueException) when calling the
CreateEventSourceMapping operation: Queue visibility timeout: 30 seconds is less
than Function timeout: 900 seconds

That validation never runs again. Raise the function timeout from 60 s to 900 s six months later — a one-line change in your template — and nothing complains, because the mapping already exists. Assert the relationship in CI instead of trusting the API to catch it.

The bucket refuses to save the notification #

An error occurred (InvalidArgument) when calling the
PutBucketNotificationConfiguration operation: Unable to validate the following
destination configurations

This single message covers three distinct causes: the queue policy does not allow s3.amazonaws.com to SendMessage, the queue is in a different region from the bucket, or the destination is a FIFO queue. S3 performs a test write when you save the configuration, and any failure produces the same text. Check the policy first — it is the cause about nine times in ten — and remember the successful test write appears on your queue as an s3:TestEvent message, which the handler above acknowledges rather than treating as a malformed job.

Throttling looks exactly like failure #

If the function is throttled, the batch is never processed, but the receive still counted. With maxReceiveCount: 5 and a concurrency limit that is regularly hit, healthy messages walk to the dead-letter queue purely because they arrived during a spike. The symptom is a dead-letter queue full of jobs that succeed instantly when you redrive them. Fix it with ScalingConfig.MaximumConcurrency rather than function-level reserved concurrency: the mapping then simply polls more slowly instead of invoking and failing. The client-side analogue is the same reasoning behind implementing exponential backoff for failed chunks.

The function times out mid-transcode #

2026-07-26T09:14:02.118Z 8f3c1e2a-... Task timed out after 900.09 seconds

Lambda kills the sandbox with no chance to return batchItemFailures, so the whole batch replays — including messages that had already finished. This is the strongest argument for small batches on long jobs. Add your own guard: compare ctx.getRemainingTimeInMillis() against your p99 job duration before starting each record in the loop, and report the remainder as failures rather than starting work you cannot finish.

Verification #

First, confirm the mapping is actually configured the way you think. This is the check that catches the silent partial-batch bug:

UUID=$(aws lambda list-event-source-mappings \
  --function-name transcode-worker \
  --query 'EventSourceMappings[0].UUID' --output text)

aws lambda get-event-source-mapping --uuid "$UUID" \
  --query '{types:FunctionResponseTypes,batch:BatchSize,state:State}'
# Expect: {"types": ["ReportBatchItemFailures"], "batch": 5, "state": "Enabled"}

Then prove the timeout arithmetic holds, comparing the queue against the function:

VT=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names VisibilityTimeout \
  --query 'Attributes.VisibilityTimeout' --output text)
FT=$(aws lambda get-function-configuration --function-name transcode-worker \
  --query Timeout --output text)

[ "$VT" -ge $(( FT * 6 )) ] || { echo "FAIL: VT=$VT needs $(( FT * 6 ))"; exit 1; }

Finally, force a redelivery and watch it stay isolated. Send one message that you know the worker rejects, then poll the depths — ApproximateNumberOfMessagesNotVisible is in-flight work, and it should return to zero while the dead-letter queue gains exactly one message after five attempts:

aws sqs send-message --queue-url "$QUEUE_URL" \
  --message-body '{"Records":[{"eventName":"ObjectCreated:Put","s3":{"bucket":{"name":"acme-uploads"},"object":{"key":"broken.mp4","size":12,"eTag":"d41d8cd98f00b204e9800998ecf8427e"}}}]}'

aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages

Alarm on ApproximateAgeOfOldestMessage on the main queue rather than its depth: depth is meaningless during a legitimate burst, whereas an oldest message older than fifteen minutes means the drain rate has lost. Record job outcomes in the same table that holds your asset rows so a stalled backlog is queryable — see how to index file metadata in PostgreSQL for the schema this attaches to.

Frequently Asked Questions