Setting Up S3 Lifecycle Rules for Temporary Uploads #

Put every browser upload into a staging/ prefix, attach one Expiration.Days rule scoped to that prefix, and make promotion a CopyObject out of it — the rule then deletes exactly the objects nobody claimed, and nothing else.

This is the concrete recipe for a staging bucket. The mechanics of the evaluation engine — midnight rounding, rule union semantics, transition economics — live on the parent guide to cloud storage lifecycle rules inside backend validation and cloud storage architecture. What follows assumes you have read that and now need a policy you can deploy on Monday.

When to use this approach #

• You issue presigned URLs and cannot see the outcome. With presigned URL workflows the browser talks to S3 directly, so your server never learns that a user closed the tab at 80%. A lifecycle rule is the only cleanup that does not depend on a client callback arriving.
• Promotion is already a copy, not a database flag. If a validated object physically moves from staging/ to originals/, the prefix boundary carries the whole retention decision. If promotion only flips a column, stop here and fix that first — a lifecycle rule cannot tell live data from debris by reading your database.
• Sub-hour precision is not required. Days is the finest granularity S3 offers, and the real window is wider than the number suggests. If you need “gone within 30 minutes”, you need a scheduled sweeper, not this.

Against the alternatives: an EventBridge-scheduled Lambda that walks ListObjectsV2 gives you minute-level precision but costs a LIST request per 1,000 keys per run and has to be monitored like any other job. Deleting from the application after processing is free but never fires for the transfers that failed — which is the entire population you are trying to clean up.

Prerequisites #

1. Node 20+ with @aws-sdk/client-s3 3.600 or later. The examples are ESM with top-level await; add "type": "module" to package.json.
2. AWS CLI v2.15+ for the verification commands.
3. A bucket whose keys are already split by state — staging/, originals/, derivatives/. If yours are flat, changing the key scheme is a prerequisite, not an optimisation.
4. An IAM principal with the bucket-level lifecycle permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManageLifecycle",
      "Effect": "Allow",
      "Action": ["s3:GetLifecycleConfiguration", "s3:PutLifecycleConfiguration"],
      "Resource": "arn:aws:s3:::media-uploads"
    },
    {
      "Sid": "PromoteAndProbe",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:PutObjectTagging"],
      "Resource": "arn:aws:s3:::media-uploads/*"
    }
  ]
}

The three kinds of debris a staging prefix accumulates #

A staging prefix does not fill up with one kind of junk, and a single rule will not clear all of it. Completed objects that nobody promoted are visible in a listing and billed normally. Abandoned multipart parts are billed but invisible to ListObjectsV2. Rejected uploads — the ones that failed server-side file validation or came back positive from a scanner — are visible, known-bad, and should not wait out the full window.

The abort action is the half most teams miss, and it has enough depth of its own to warrant a separate treatment in expiring incomplete multipart uploads automatically. Include it in the policy below, then read that page to size DaysAfterInitiation against your slowest genuine session.

Sizing the expiry window #

The number you put in Days is not the lifetime of the object. S3 rounds the creation timestamp up to the next UTC midnight, adds Days, and only then queues the delete — with up to 48 hours of execution lag after eligibility. Two consequences fall out, and you need both:

• Guaranteed minimum life is Days × 24 hours. The worst case for your pipeline is an object created a second before midnight, where the rounding adds nothing. Your processing budget must fit inside that.
• Worst-case maximum life is (Days + 1) × 24 + 48 hours. The worst case for your storage bill is an object created a second after midnight that then waits out the full execution lag. Your cost model must assume that.

Measure the budget rather than guessing it. The inputs are the p99 wall-clock time from PutObject to promotion, plus whatever manual requeue window your on-call rotation needs. For a pipeline that runs a virus scan and one transcode pass through SQS and Lambda, 8 hours of p99 plus 22 hours of retry budget is realistic; that lands on Days: 2, a guaranteed 48 hours of survival and a worst case of 120.

Implementation #

Build the policy from a typed function so the window and the rule stay in sync — hand-edited JSON is where the mismatch between “we expire after two days” and “we actually expire after one” comes from.

// staging-lifecycle.ts
import type { LifecycleRule } from "@aws-sdk/client-s3";

export interface StagingWindow {
  /** Prefix the browser uploads land in. Must end with a slash. */
  prefix: string;
  /** p99 wall-clock hours from PutObject to promotion. Measured, not guessed. */
  pipelineHours: number;
  /** Hours of retry and manual-requeue budget on top of the pipeline. */
  retryHours: number;
}

const HOURS_PER_DAY = 24;
const EXECUTION_LAG_HOURS = 48;

/** Guaranteed minimum survival is Days * 24h, so round the budget up to whole days. */
export function expirationDays(spec: StagingWindow): number {
  const budget = spec.pipelineHours + spec.retryHours;
  if (budget <= 0) throw new RangeError("processing budget must be positive");
  return Math.max(1, Math.ceil(budget / HOURS_PER_DAY));
}

/** Worst-case billed lifetime, for the cost model. */
export function worstCaseHours(days: number): number {
  return (days + 1) * HOURS_PER_DAY + EXECUTION_LAG_HOURS;
}

export function stagingPolicy(spec: StagingWindow): LifecycleRule[] {
  if (!spec.prefix.endsWith("/")) {
    throw new Error(`prefix ${JSON.stringify(spec.prefix)} must end with "/" — S3 prefix matching is literal`);
  }
  const days = expirationDays(spec);
  return [
    {
      ID: "expire-staging-objects",
      Status: "Enabled",
      Filter: { Prefix: spec.prefix },
      Expiration: { Days: days },
    },
    {
      ID: "abort-staging-parts",
      Status: "Enabled",
      Filter: { Prefix: spec.prefix },
      AbortIncompleteMultipartUpload: { DaysAfterInitiation: days },
    },
    {
      // Known-bad uploads do not deserve the full window. Tags need an And block.
      ID: "expire-rejected-uploads",
      Status: "Enabled",
      Filter: { And: { Prefix: spec.prefix, Tags: [{ Key: "state", Value: "rejected" }] } },
      Expiration: { Days: 1 },
    },
  ];
}

Four parameters do the real work here:

• Filter.Prefix: "staging/" — the trailing slash is load-bearing. S3 matches prefixes as literal byte strings with no notion of directories, so staging without the slash also matches staging-archive/2024/report.csv. The endsWith guard exists because that mistake deletes a different prefix entirely and you find out 48 hours later.
• Expiration.Days — a positive integer only. There is no hours field, no Expiration.Hours, and no way to express “expire at 04:00”.
• AbortIncompleteMultipartUpload.DaysAfterInitiation — kept in a rule of its own rather than merged into the first. S3 accepts both actions in one rule, but separate IDs mean the s3:LifecycleExpiration:* audit events and any CloudWatch alarm you build can tell an abandoned part from an unclaimed object.
• Filter.And.Tags — a rule may carry Prefix or Tag at the top level, never both. Combining them requires the And wrapper, and the tag must be an object tag, not user metadata; x-amz-meta-* is invisible to the lifecycle engine.

Deploying is a read-modify-write, because PutBucketLifecycleConfiguration replaces the whole document with no append and no conditional header. The merge-by-ID guard is explained in full on the parent guide; the staging-specific part is ExpectedBucketOwner, which turns a mistyped bucket name in a shared account from a silent policy overwrite into a 403.

// deploy-staging-lifecycle.ts
import {
  S3Client,
  GetBucketLifecycleConfigurationCommand,
  PutBucketLifecycleConfigurationCommand,
  type LifecycleRule,
} from "@aws-sdk/client-s3";
import { stagingPolicy, expirationDays, worstCaseHours } from "./staging-lifecycle.js";

const BUCKET = process.env.BUCKET ?? "media-uploads";
const OWNER = process.env.AWS_ACCOUNT_ID ?? "111122223333";
const client = new S3Client({ region: process.env.AWS_REGION ?? "eu-west-1", maxAttempts: 5 });

const spec = { prefix: "staging/", pipelineHours: 8, retryHours: 22 };
const desired = stagingPolicy(spec);

let live: LifecycleRule[] = [];
try {
  const res = await client.send(
    new GetBucketLifecycleConfigurationCommand({ Bucket: BUCKET, ExpectedBucketOwner: OWNER }),
  );
  live = res.Rules ?? [];
} catch (error) {
  if ((error as { name?: string }).name !== "NoSuchLifecycleConfiguration") throw error;
}

const byId = new Map(live.map((rule) => [rule.ID ?? "", rule]));
const ourIds = new Set(desired.map((rule) => rule.ID ?? ""));
for (const rule of desired) byId.set(rule.ID ?? "", rule);
const merged = [...byId.values()];

await client.send(
  new PutBucketLifecycleConfigurationCommand({
    Bucket: BUCKET,
    ExpectedBucketOwner: OWNER,
    LifecycleConfiguration: { Rules: merged },
  }),
);

const days = expirationDays(spec);
console.log(
  `${BUCKET}: ${merged.length} rule(s) live, ${ourIds.size} owned here, ` +
    `${merged.length - ourIds.size} preserved. Guaranteed ${days * 24}h, worst case ${worstCaseHours(days)}h.`,
);

A first run against a bucket that already carries an unrelated archival rule prints:

media-uploads: 4 rule(s) live, 3 owned here, 1 preserved. Guaranteed 48h, worst case 120h.

Promoting an object out of the window #

The rule only stays safe if promotion physically leaves the prefix. CopyObject is a server-side operation — no bytes traverse your process — and it is the moment to replace the staging tags, because TaggingDirective: "COPY" would carry state=rejected straight into your permanent prefix.

// promote.ts
import {
  S3Client,
  CopyObjectCommand,
  DeleteObjectCommand,
  HeadObjectCommand,
} from "@aws-sdk/client-s3";

const BUCKET = process.env.BUCKET ?? "media-uploads";
const client = new S3Client({ region: process.env.AWS_REGION ?? "eu-west-1" });

/** Encode each path segment: keys legitimately contain spaces, '+' and '#'. */
function copySource(bucket: string, key: string): string {
  return `/${bucket}/${key.split("/").map(encodeURIComponent).join("/")}`;
}

export async function promote(stagingKey: string, finalKey: string): Promise<string> {
  const copied = await client.send(
    new CopyObjectCommand({
      Bucket: BUCKET,
      Key: finalKey,
      CopySource: copySource(BUCKET, stagingKey),
      MetadataDirective: "COPY",
      TaggingDirective: "REPLACE",
      Tagging: "state=live",
      ChecksumAlgorithm: "SHA256",
    }),
  );

  // The destination must not match any expiration rule. If it does, the prefix is wrong.
  const head = await client.send(new HeadObjectCommand({ Bucket: BUCKET, Key: finalKey }));
  if (head.Expiration) {
    await client.send(new DeleteObjectCommand({ Bucket: BUCKET, Key: finalKey }));
    throw new Error(`refusing to promote into an expiring prefix: ${head.Expiration}`);
  }

  await client.send(new DeleteObjectCommand({ Bucket: BUCKET, Key: stagingKey }));
  return copied.CopyObjectResult?.ETag ?? "";
}

Two limits to plan around. CopyObject refuses anything over 5 GB with InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 — above that you need a multipart copy using UploadPartCopy, which is a different code path and worth testing before a customer finds it. And the explicit DeleteObject after the copy is optional: leaving the staging object in place lets the lifecycle rule collect it, which costs a day of duplicate storage but keeps a free rollback while the promotion is still fresh. Record the new key in the same transaction that updates your index, following the schema in how to index file metadata in PostgreSQL.

Configuration gotchas #

MalformedXML from a prefix and a tag in the same filter #

Writing Filter: { Prefix: "staging/", Tag: { Key: "state", Value: "rejected" } } fails with An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema. The schema allows exactly one condition at the top level of Filter. Wrap two or more in And, and note that And requires at least two conditions — an And containing only a prefix is rejected by the same error.

InvalidRequest on a zero or fractional window #

Expiration: { Days: 0 } returns InvalidRequest: 'Days' for Expiration action must be a positive integer. The same applies to DaysAfterInitiation. There is no sub-day expiration in S3 lifecycle at all, so if a compliance rule says “delete within 6 hours” the honest answer is a scheduled job, and the lifecycle rule becomes the backstop behind it rather than the mechanism.

The presigned URL that outlives its own object #

An upload URL signed with a 12-hour expiresIn against a prefix that expires in one day leaves almost no margin: a client that starts the transfer at hour 11, retries through a bad connection, and finishes at hour 20 has written an object that may already be eligible. Keep expiresIn at minutes, not hours, and re-sign on resume — the pattern described in resuming uploads after network loss. A stale URL fails loudly with 403 AccessDenied: Request has expired, which is far easier to debug than an object that quietly disappeared.

Versioning turns the delete into a delete marker #

On a versioned bucket, Expiration.Days does not free any bytes: it writes a delete marker and moves the old version to noncurrent, where it is still billed. A staging prefix on a versioned bucket therefore needs NoncurrentVersionExpiration: { NoncurrentDays: 1 } alongside the expiration, and a separate rule with Expiration: { ExpiredObjectDeleteMarker: true } to sweep the markers left behind. ExpiredObjectDeleteMarker cannot share a rule with Days or Date — that combination is another MalformedXML.

Verification #

There is no dry-run endpoint, but there is something better for this specific case: after the rule is live, S3 returns an x-amz-expiration header on every PutObject and HeadObject response for a matching key, telling you the exact instant and the exact rule ID that owns it.

Three checks, in order. First, confirm a staging key is covered:

aws s3api head-object --bucket media-uploads \
  --key staging/9f2c41/clip.mov --query 'Expiration' --output text

expiry-date="Sat, 28 Mar 2026 00:00:00 GMT", rule-id="expire-staging-objects"

Second, confirm the filter does not over-match. Write a probe just outside the prefix and check that the header is absent — this is the trailing-slash bug catching itself in one command:

aws s3api put-object --bucket media-uploads --key staging-archive/probe.bin \
  --body /dev/null --query 'Expiration' --output text

None

Third, assert it continuously from the application. HeadObjectCommand surfaces the same header as Expiration, so a test that runs after each deploy is four lines:

// assert-coverage.ts
import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";
import assert from "node:assert/strict";

const client = new S3Client({ region: process.env.AWS_REGION ?? "eu-west-1" });
const BUCKET = process.env.BUCKET ?? "media-uploads";

const staged = await client.send(new HeadObjectCommand({ Bucket: BUCKET, Key: "staging/probe.bin" }));
assert.match(staged.Expiration ?? "", /rule-id="expire-staging-objects"/, "staging key is not covered");

const live = await client.send(new HeadObjectCommand({ Bucket: BUCKET, Key: "originals/probe.bin" }));
assert.equal(live.Expiration, undefined, `promoted key is still expiring: ${live.Expiration}`);
console.log("lifecycle coverage verified for staging/ and originals/");

Beyond the header, enable the s3:LifecycleExpiration:Delete event notification and alarm on the daily count going to zero. A rule that stops matching — because someone changed the key scheme — is silent, and the first symptom is a storage bill three months later.

Frequently Asked Questions