Building a Production-Grade Object Storage Migration Tool Using Node.js and AWS S3

Introduction

Organizations frequently need to migrate files and objects between different cloud storage providers. This migration may occur due to cloud adoption, cost optimization, infrastructure modernization, or application migration.

Recently, we implemented a production-ready object migration solution using Node.js to transfer files from an S3-compatible object storage service to Amazon S3.

This article explains the architecture, implementation approach, challenges, and production considerations involved in building a scalable migration utility.

Problem Statement

The source storage platform provided an S3-compatible API, while the destination storage was Amazon S3.

The migration solution needed to support:

Thousands of files
Folder structure preservation
Large file handling
Retry mechanisms
Resume capability
Concurrent uploads
Logging and monitoring
Production reliability

Architecture

The migration flow is straightforward:

Source Object Storage
        ↓ Download Object
        ↓ Process Stream
        ↓ Upload Object
        ↓ Amazon S3

The application performs the following operations:

List all objects from the source bucket.
Download each object.
Upload the object to AWS S3.
Preserve the original object key.
Track migration status.

Technology Stack

Node.js
AWS SDK v3
Amazon S3
S3-Compatible Object Storage
dotenv
p-limit
AWS Multipart Upload

Key Features

1. Concurrent Processing

Instead of uploading files one by one, multiple files are processed simultaneously.

Benefits

Faster migration
Better network utilization
Reduced execution time

2. Retry Mechanism

Temporary network failures can occur during large migrations.

The application retries failed uploads multiple times before marking them as failed.

Benefits

Increased reliability
Reduced manual intervention

3. Resume Support

Migration can resume from the last successful state.

Previously migrated files are skipped automatically.

Benefits

No duplicate uploads
Faster recovery

4. Success and Failure Logs

Two log files are maintained:

success.log
failed.log

Benefits

Easy auditing
Failure analysis
Selective reprocessing

5. Skip Existing Objects

Before uploading, the application checks whether the object already exists in the destination bucket.

Benefits

Avoids duplicate uploads
Reduces API calls

6. Multipart Upload

Large files require special handling.

Multipart upload allows:

Parallel part uploads
Better reliability
Reduced memory usage

This is especially important for:

Videos
Backups
Archives
Large media files

Challenges Faced

Missing Content Length

The source object storage did not return certain metadata required by AWS SDK.

This caused:

Invalid value "undefined" for header "x-amz-decoded-content-length"

Solution

The object stream was converted into a buffer before upload.

Temporary AWS Credentials

The AWS environment provided temporary STS credentials.

These credentials required:

Access Key
Secret Key
Session Token

Without the session token, uploads failed with:

InvalidAccessKeyId

Dependency Compatibility

The p-limit package introduced ESM changes in newer versions.

Solution

Use the appropriate import syntax based on the installed version.

Production Considerations

For enterprise-scale migrations:

Use multipart uploads.
Implement monitoring dashboards.
Store migration status in a database.
Add progress reporting.
Introduce alerting mechanisms.
Use IAM roles instead of static credentials.
Implement checksum validation.

Security Best Practices

Never hardcode credentials.
Store secrets in environment variables.
Use IAM policies with minimum permissions.
Rotate credentials periodically.
Use temporary credentials where possible.

Performance Improvements

Potential future enhancements:

Dynamic concurrency tuning
Distributed workers
Queue-based processing
Database-backed checkpoints
Object integrity validation

Code

require("dotenv").config();

const fs = require("fs");
const pLimit = require("p-limit");

const {
  S3Client,
  ListObjectsV2Command,
  GetObjectCommand,
  HeadObjectCommand,
  PutObjectCommand,
} = require("@aws-sdk/client-s3");

const { Upload } = require("@aws-sdk/lib-storage");

const limit = pLimit(
  Number(process.env.CONCURRENCY || 5)
);

const RETRY_COUNT = Number(
  process.env.RETRY_COUNT || 3
);

const MULTIPART_THRESHOLD =
  Number(process.env.MULTIPART_THRESHOLD_MB || 50) *
  1024 *
  1024;

const successLog = "success.log";
const failedLog = "failed.log";

const sourceS3 = new S3Client({
  endpoint: process.env.NEEDCLOUD_ENDPOINT,
  region: "us-east-1",
  forcePathStyle: true,
  credentials: {
    accessKeyId:
      process.env.NEEDCLOUD_ACCESS_KEY,
    secretAccessKey:
      process.env.NEEDCLOUD_SECRET_KEY,
  },
});

const destinationS3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId:
      process.env.AWS_ACCESS_KEY,
    secretAccessKey:
      process.env.AWS_SECRET_KEY,
    sessionToken:
      process.env.AWS_SESSION_TOKEN,
  },
});

const SOURCE_BUCKET =
  process.env.SOURCE_BUCKET;

const DESTINATION_BUCKET =
  process.env.DESTINATION_BUCKET;

function logSuccess(key) {
  fs.appendFileSync(successLog, key + "\n");
}

function logFailure(key, error) {
  fs.appendFileSync(
    failedLog,
    `${key} | ${error}\n`
  );
}

const completed = new Set();

if (fs.existsSync(successLog)) {
  fs.readFileSync(successLog, "utf8")
    .split("\n")
    .filter(Boolean)
    .forEach((x) => completed.add(x));
}

async function streamToBuffer(stream) {
  const chunks = [];

  for await (const chunk of stream) {
    chunks.push(
      chunk instanceof Buffer
        ? chunk
        : Buffer.from(chunk)
    );
  }

  return Buffer.concat(chunks);
}

async function exists(key) {
  try {
    await destinationS3.send(
      new HeadObjectCommand({
        Bucket: DESTINATION_BUCKET,
        Key: key,
      })
    );
    return true;
  } catch {
    return false;
  }
}

async function uploadSmallFile(
  key,
  buffer,
  contentType
) {
  await destinationS3.send(
    new PutObjectCommand({
      Bucket: DESTINATION_BUCKET,
      Key: key,
      Body: buffer,
      ContentType:
        contentType ||
        "application/octet-stream",
    })
  );
}

async function uploadLargeFile(
  key,
  body,
  contentType
) {
  const upload = new Upload({
    client: destinationS3,
    params: {
      Bucket: DESTINATION_BUCKET,
      Key: key,
      Body: body,
      ContentType:
        contentType ||
        "application/octet-stream",
    },
    queueSize: 4,
    partSize: 5 * 1024 * 1024,
  });

  await upload.done();
}

async function copyFile(key) {
  if (completed.has(key)) {
    console.log(`SKIP: ${key}`);
    return;
  }

  if (await exists(key)) {
    console.log(`ALREADY EXISTS: ${key}`);
    logSuccess(key);
    return;
  }

  for (
    let attempt = 1;
    attempt <= RETRY_COUNT;
    attempt++
  ) {
    try {
      console.log(
        `[${attempt}] Downloading ${key}`
      );

      const response =
        await sourceS3.send(
          new GetObjectCommand({
            Bucket: SOURCE_BUCKET,
            Key: key,
          })
        );

      const size =
        response.ContentLength || 0;

      if (size > MULTIPART_THRESHOLD) {
        console.log(
          `Multipart Upload: ${key}`
        );

        await uploadLargeFile(
          key,
          response.Body,
          response.ContentType
        );
      } else {
        const buffer =
          await streamToBuffer(
            response.Body
          );

        await uploadSmallFile(
          key,
          buffer,
          response.ContentType
        );
      }

      console.log(`SUCCESS: ${key}`);

      logSuccess(key);

      return;
    } catch (err) {
      console.error(
        `Attempt ${attempt} failed: ${key}`
      );

      if (attempt === RETRY_COUNT) {
        logFailure(
          key,
          err.message
        );
      }
    }
  }
}

async function getAllFiles() {
  let token;
  const files = [];

  do {
    const result =
      await sourceS3.send(
        new ListObjectsV2Command({
          Bucket: SOURCE_BUCKET,
          ContinuationToken: token,
        })
      );

    files.push(
      ...(result.Contents || [])
    );

    token =
      result.NextContinuationToken;
  } while (token);

  return files;
}

async function migrate() {
  const files = await getAllFiles();

  console.log(
    `Total Files: ${files.length}`
  );

  let processed = 0;

  await Promise.all(
    files.map((file) =>
      limit(async () => {
        await copyFile(file.Key);

        processed++;

        console.log(
          `Progress: ${processed}/${files.length}`
        );
      })
    )
  );

  console.log("Migration Completed");
}

migrate().catch(console.error);

Conclusion

Building a production-grade object migration utility requires much more than simply copying files between two buckets.

Reliability, scalability, retry mechanisms, logging, concurrency, security, and resumability are critical factors in any real-world migration system.

By combining Node.js, AWS SDK v3, and modern cloud practices, it is possible to build a robust migration framework capable of handling thousands of objects efficiently and reliably.

This approach provides a flexible and scalable foundation for migrating data between any S3-compatible object storage platform and Amazon S3.