Cloud  

Building a Production-Grade Object Storage Migration Tool Using Node.js and AWS S3

Introduction

Organizations frequently need to migrate files and objects between different cloud storage providers. This migration may occur due to cloud adoption, cost optimization, infrastructure modernization, or application migration.

Recently, we implemented a production-ready object migration solution using Node.js to transfer files from an S3-compatible object storage service to Amazon S3.

This article explains the architecture, implementation approach, challenges, and production considerations involved in building a scalable migration utility.

Problem Statement

The source storage platform provided an S3-compatible API, while the destination storage was Amazon S3.

The migration solution needed to support:

  • Thousands of files

  • Folder structure preservation

  • Large file handling

  • Retry mechanisms

  • Resume capability

  • Concurrent uploads

  • Logging and monitoring

  • Production reliability

Architecture

The migration flow is straightforward:

Source Object Storage
        ↓ Download Object
        ↓ Process Stream
        ↓ Upload Object
        ↓ Amazon S3

The application performs the following operations:

  1. List all objects from the source bucket.

  2. Download each object.

  3. Upload the object to AWS S3.

  4. Preserve the original object key.

  5. Track migration status.

Technology Stack

  • Node.js

  • AWS SDK v3

  • Amazon S3

  • S3-Compatible Object Storage

  • dotenv

  • p-limit

  • AWS Multipart Upload

Key Features

1. Concurrent Processing

Instead of uploading files one by one, multiple files are processed simultaneously.

Benefits

  • Faster migration

  • Better network utilization

  • Reduced execution time

2. Retry Mechanism

Temporary network failures can occur during large migrations.

The application retries failed uploads multiple times before marking them as failed.

Benefits

  • Increased reliability

  • Reduced manual intervention

3. Resume Support

Migration can resume from the last successful state.

Previously migrated files are skipped automatically.

Benefits

  • No duplicate uploads

  • Faster recovery

4. Success and Failure Logs

Two log files are maintained:

  • success.log

  • failed.log

Benefits

  • Easy auditing

  • Failure analysis

  • Selective reprocessing

5. Skip Existing Objects

Before uploading, the application checks whether the object already exists in the destination bucket.

Benefits

  • Avoids duplicate uploads

  • Reduces API calls

6. Multipart Upload

Large files require special handling.

Multipart upload allows:

  • Parallel part uploads

  • Better reliability

  • Reduced memory usage

This is especially important for:

  • Videos

  • Backups

  • Archives

  • Large media files

Challenges Faced

Missing Content Length

The source object storage did not return certain metadata required by AWS SDK.

This caused:

Invalid value "undefined" for header "x-amz-decoded-content-length"

Solution

The object stream was converted into a buffer before upload.

Temporary AWS Credentials

The AWS environment provided temporary STS credentials.

These credentials required:

  • Access Key

  • Secret Key

  • Session Token

Without the session token, uploads failed with:

InvalidAccessKeyId

Dependency Compatibility

The p-limit package introduced ESM changes in newer versions.

Solution

Use the appropriate import syntax based on the installed version.

Production Considerations

For enterprise-scale migrations:

  • Use multipart uploads.

  • Implement monitoring dashboards.

  • Store migration status in a database.

  • Add progress reporting.

  • Introduce alerting mechanisms.

  • Use IAM roles instead of static credentials.

  • Implement checksum validation.

Security Best Practices

  • Never hardcode credentials.

  • Store secrets in environment variables.

  • Use IAM policies with minimum permissions.

  • Rotate credentials periodically.

  • Use temporary credentials where possible.

Performance Improvements

Potential future enhancements:

  • Dynamic concurrency tuning

  • Distributed workers

  • Queue-based processing

  • Database-backed checkpoints

  • Object integrity validation

Code

require("dotenv").config();

const fs = require("fs");
const pLimit = require("p-limit");

const {
  S3Client,
  ListObjectsV2Command,
  GetObjectCommand,
  HeadObjectCommand,
  PutObjectCommand,
} = require("@aws-sdk/client-s3");

const { Upload } = require("@aws-sdk/lib-storage");

const limit = pLimit(
  Number(process.env.CONCURRENCY || 5)
);

const RETRY_COUNT = Number(
  process.env.RETRY_COUNT || 3
);

const MULTIPART_THRESHOLD =
  Number(process.env.MULTIPART_THRESHOLD_MB || 50) *
  1024 *
  1024;

const successLog = "success.log";
const failedLog = "failed.log";

const sourceS3 = new S3Client({
  endpoint: process.env.NEEDCLOUD_ENDPOINT,
  region: "us-east-1",
  forcePathStyle: true,
  credentials: {
    accessKeyId:
      process.env.NEEDCLOUD_ACCESS_KEY,
    secretAccessKey:
      process.env.NEEDCLOUD_SECRET_KEY,
  },
});

const destinationS3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId:
      process.env.AWS_ACCESS_KEY,
    secretAccessKey:
      process.env.AWS_SECRET_KEY,
    sessionToken:
      process.env.AWS_SESSION_TOKEN,
  },
});

const SOURCE_BUCKET =
  process.env.SOURCE_BUCKET;

const DESTINATION_BUCKET =
  process.env.DESTINATION_BUCKET;

function logSuccess(key) {
  fs.appendFileSync(successLog, key + "\n");
}

function logFailure(key, error) {
  fs.appendFileSync(
    failedLog,
    `${key} | ${error}\n`
  );
}

const completed = new Set();

if (fs.existsSync(successLog)) {
  fs.readFileSync(successLog, "utf8")
    .split("\n")
    .filter(Boolean)
    .forEach((x) => completed.add(x));
}

async function streamToBuffer(stream) {
  const chunks = [];

  for await (const chunk of stream) {
    chunks.push(
      chunk instanceof Buffer
        ? chunk
        : Buffer.from(chunk)
    );
  }

  return Buffer.concat(chunks);
}

async function exists(key) {
  try {
    await destinationS3.send(
      new HeadObjectCommand({
        Bucket: DESTINATION_BUCKET,
        Key: key,
      })
    );
    return true;
  } catch {
    return false;
  }
}

async function uploadSmallFile(
  key,
  buffer,
  contentType
) {
  await destinationS3.send(
    new PutObjectCommand({
      Bucket: DESTINATION_BUCKET,
      Key: key,
      Body: buffer,
      ContentType:
        contentType ||
        "application/octet-stream",
    })
  );
}

async function uploadLargeFile(
  key,
  body,
  contentType
) {
  const upload = new Upload({
    client: destinationS3,
    params: {
      Bucket: DESTINATION_BUCKET,
      Key: key,
      Body: body,
      ContentType:
        contentType ||
        "application/octet-stream",
    },
    queueSize: 4,
    partSize: 5 * 1024 * 1024,
  });

  await upload.done();
}

async function copyFile(key) {
  if (completed.has(key)) {
    console.log(`SKIP: ${key}`);
    return;
  }

  if (await exists(key)) {
    console.log(`ALREADY EXISTS: ${key}`);
    logSuccess(key);
    return;
  }

  for (
    let attempt = 1;
    attempt <= RETRY_COUNT;
    attempt++
  ) {
    try {
      console.log(
        `[${attempt}] Downloading ${key}`
      );

      const response =
        await sourceS3.send(
          new GetObjectCommand({
            Bucket: SOURCE_BUCKET,
            Key: key,
          })
        );

      const size =
        response.ContentLength || 0;

      if (size > MULTIPART_THRESHOLD) {
        console.log(
          `Multipart Upload: ${key}`
        );

        await uploadLargeFile(
          key,
          response.Body,
          response.ContentType
        );
      } else {
        const buffer =
          await streamToBuffer(
            response.Body
          );

        await uploadSmallFile(
          key,
          buffer,
          response.ContentType
        );
      }

      console.log(`SUCCESS: ${key}`);

      logSuccess(key);

      return;
    } catch (err) {
      console.error(
        `Attempt ${attempt} failed: ${key}`
      );

      if (attempt === RETRY_COUNT) {
        logFailure(
          key,
          err.message
        );
      }
    }
  }
}

async function getAllFiles() {
  let token;
  const files = [];

  do {
    const result =
      await sourceS3.send(
        new ListObjectsV2Command({
          Bucket: SOURCE_BUCKET,
          ContinuationToken: token,
        })
      );

    files.push(
      ...(result.Contents || [])
    );

    token =
      result.NextContinuationToken;
  } while (token);

  return files;
}

async function migrate() {
  const files = await getAllFiles();

  console.log(
    `Total Files: ${files.length}`
  );

  let processed = 0;

  await Promise.all(
    files.map((file) =>
      limit(async () => {
        await copyFile(file.Key);

        processed++;

        console.log(
          `Progress: ${processed}/${files.length}`
        );
      })
    )
  );

  console.log("Migration Completed");
}

migrate().catch(console.error);

Conclusion

Building a production-grade object migration utility requires much more than simply copying files between two buckets.

Reliability, scalability, retry mechanisms, logging, concurrency, security, and resumability are critical factors in any real-world migration system.

By combining Node.js, AWS SDK v3, and modern cloud practices, it is possible to build a robust migration framework capable of handling thousands of objects efficiently and reliably.

This approach provides a flexible and scalable foundation for migrating data between any S3-compatible object storage platform and Amazon S3.

Membership not found