Implementing ClamAV for Uploaded File Scanning in Direct-to-Cloud Workflows #

Securing user-generated content requires a robust Backend Validation & Cloud Storage Architecture that decouples ingestion from processing. Direct-to-cloud uploads via presigned URLs eliminate backend bottlenecks. They also introduce blind spots in malware detection. This guide details how to implement asynchronous ClamAV scanning. You will maintain zero-latency client experiences while enforcing enterprise-grade defense.

Key architectural principles:

• Decouple upload ingestion from malware scanning using event-driven architectures.
• Deploy ClamAV in ephemeral compute environments with mounted signature volumes.
• Enforce strict quarantine workflows and metadata state transitions for infected payloads.

Architecting the Async Scan Pipeline #

The event flow must guarantee exactly-once processing semantics. Configure S3 to emit ObjectCreated:CompleteMultipartUpload events directly to an SQS FIFO queue. This prevents premature scans triggered by incomplete multipart uploads.

Apply an S3 object tag (scan_status=pending) before dispatching the event. This acts as a distributed lock. It prevents race conditions when multiple consumers poll the queue.

// AWS SDK v3 - S3 Tagging & SQS Dispatch
import { S3Client, PutObjectTaggingCommand } from "@aws-sdk/client-s3";
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const s3 = new S3Client({ region: process.env.AWS_REGION });
const sqs = new SQSClient({ region: process.env.AWS_REGION });

export async function triggerAsyncScan(event) {
 const { bucket, key, etag } = event.Records[0].s3;
 const decodedKey = decodeURIComponent(key.replace(/\+/g, " "));

 try {
 await s3.send(new PutObjectTaggingCommand({
 Bucket: bucket.name,
 Key: decodedKey,
 Tagging: { TagSet: [{ Key: "scan_status", Value: "pending" }] }
 }));

 await sqs.send(new SendMessageCommand({
 QueueUrl: process.env.SCAN_QUEUE_URL,
 MessageBody: JSON.stringify({ bucket: bucket.name, key: decodedKey, etag }),
 MessageGroupId: "clamav-scan-group",
 MessageDeduplicationId: `${decodedKey}-${etag}`
 }));
 } catch (err) {
 console.error("Pipeline trigger failed:", err);
 throw new Error("Failed to enqueue scan job");
 }
}

Idempotency relies on MessageDeduplicationId and SQS visibility timeouts. Consumers must verify the scan_status tag before initiating the scan. This gracefully handles duplicate triggers and network retries.

Containerizing ClamAV for Serverless Execution #

Running clamd in AWS Lambda requires strict resource isolation. The daemon must operate within constrained memory limits. It must also maintain fresh signature definitions.

Use a multi-stage Dockerfile that compiles ClamAV from source. Configure clamd for single-threaded execution. This prevents out-of-memory (OOM) kills during cold starts.

FROM public.ecr.aws/amazonlinux/amazonlinux:2023 AS builder
RUN dnf install -y gcc make cmake openssl-devel zlib-devel pcre2-devel json-c-devel
RUN curl -L https://www.clamav.net/downloads/production/clamav-1.2.1.tar.gz | tar xz
WORKDIR /clamav-1.2.1
RUN cmake . -DCMAKE_INSTALL_PREFIX=/opt/clamav && make -j$(nproc) && make install

FROM public.ecr.aws/lambda/node:20
COPY --from=builder /opt/clamav /opt/clamav
ENV PATH="/opt/clamav/bin:${PATH}"
ENV CLAMD_SOCKET="/tmp/clamd.sock"

RUN mkdir -p /var/lib/clamav /var/log/clamav && \
 chmod 777 /tmp && \
 freshclam --daemon-notify=/opt/clamav/etc/clamd.conf || true

COPY clamd.conf /opt/clamav/etc/clamd.conf
CMD ["handler.scan"]

The clamd.conf must explicitly limit concurrency and disable resource-heavy features:

# clamd.conf
MaxThreads 1
MaxQueue 10
StreamMaxLength 1000M
LocalSocket /tmp/clamd.sock
LogSyslog yes
LogFacility LOG_LOCAL0

Mount /var/lib/clamav via Amazon EFS to share signatures across concurrent invocations. Alternatively, download fresh signatures via freshclam during initialization. Cache them in /tmp to avoid network latency on subsequent cold starts.

Handling Scan Results & State Transitions #

The Lambda consumer streams the S3 object directly into the clamd socket. Avoid writing to /tmp to prevent ephemeral storage exhaustion. Parse the daemon’s response to determine the next state.

import { S3Client, GetObjectCommand, CopyObjectCommand, DeleteObjectCommand } from "@aws-sdk/client-s3";
import { spawn } from "child_process";
import { pipeline } from "stream/promises";

const s3 = new S3Client({ region: process.env.AWS_REGION });

export async function scanAndRoute({ bucket, key }) {
 const response = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
 const clamd = spawn("clamdscan", ["--stream", "--stdout", "--no-summary"]);

 let stdout = "";
 clamd.stdout.on("data", (chunk) => (stdout += chunk.toString()));
 clamd.stderr.on("data", (chunk) => console.error(`clamd stderr: ${chunk}`));

 const timeout = setTimeout(() => {
 clamd.kill("SIGKILL");
 throw new Error("ClamAV scan exceeded 30s timeout");
 }, 30000);

 try {
 await pipeline(response.Body, clamd.stdin);
 clearTimeout(timeout);
 } catch (err) {
 clearTimeout(timeout);
 console.error("Stream pipe failed:", err);
 throw new Error("ClamAV stream interrupted");
 }

 const result = stdout.trim();
 if (result.includes("FOUND")) {
 await quarantineObject(bucket, key);
 return { status: "infected", result };
 } else if (result.includes("OK")) {
 await updateMetadata(bucket, key, "clean");
 return { status: "clean", result };
 } else {
 throw new Error(`ClamAV returned unexpected output: ${result}`);
 }
}

async function quarantineObject(bucket, key) {
 const quarantineKey = `quarantine/${Date.now()}-${key}`;
 await s3.send(new CopyObjectCommand({
 Bucket: bucket,
 CopySource: `${bucket}/${encodeURIComponent(key)}`,
 Key: quarantineKey,
 MetadataDirective: "REPLACE",
 Metadata: { "scan-status": "quarantined", "original-key": key }
 }));
 await s3.send(new DeleteObjectCommand({ Bucket: bucket, Key: key }));
}

Implement observability hooks by emitting custom CloudWatch metrics for scan duration and queue depth. Configure S3 lifecycle rules to automatically expire objects in the quarantine/ prefix after 30 days.

Implementation Patterns & Diagnostics #

The S3 EventBridge → SQS → Lambda pattern decouples ingestion from scanning. The Lambda function downloads the object stream. It pipes data to a local clamd socket. It evaluates the response before updating DynamoDB. This ensures Automated Virus Scanning Integration operates without blocking the upload thread.

Implement a circuit breaker for ClamAV signature updates. If freshclam fails or the daemon crashes, route uploads to a fallback synchronous scanner. Reject requests with a 503 Service Unavailable if no fallback exists. Maintain a Redis-backed health check that tracks daemon uptime and signature age.

Common Pitfalls & Mitigations #

• Lambda Timeout on Large Files (>500MB): Default execution limits (15m) are insufficient for streaming large payloads through a single-threaded process. Mitigate by implementing chunked streaming with clamdscan --stream. Offload files exceeding 200MB to an ECS Fargate task with persistent EBS storage.
• Race Conditions with Presigned URL Uploads: The S3 ObjectCreated event may fire before multipart completion. Mitigate by filtering events using x-amz-meta-upload-complete headers. Rely exclusively on ObjectCreated:CompleteMultipartUpload event types.
• Stale Virus Definitions: Cold-start functions using outdated signatures fail to detect recent threats. Mitigate by scheduling a CloudWatch Event to update a shared EFS volume with freshclam every 4 hours. Bake updated .cvd files into a custom Lambda layer.

Frequently Asked Questions