Copying Data From GCP Server (Linux) To External AWS S3 Bucket

This post shows how to copy data from Linux VM in Google Cloud Platform to Amazon Web services S3 bucket.

Introduction

 

In one of my projects, I had the requirement to transfer GBs of data from one cloud platform to another cloud platform (i.e. Google Cloud Platform to Amazon Web Service S3 bucket). I had a Linux virtual machine where my data is kept in a certain directory and I have to transfer that data to some external client that has an AWS S3 bucket.

 

Options to implement the above scenario

 

Gsutil

 

You will find several articles available on the internet to transfer data from gsutil, but the problem with this approach is that you will be required to create a GCP bucket also. Gsutil transfers data from GCP bucket to AWS bucket. I have found a very useful article to get an understanding of this topic. How to transfer files from Google Cloud Storage (GCS) into Amazon S3 bucket without downloading the files - Pro Analyst.

 

We don’t want to have the extra baggage of creating a GCP bucket so I decided not to work for this approach and I had to look for other options which are shown below.

 

AWS CLI

 

Now after evaluating my first approach which as per my requirement is not the correct way. I found AWS CLI will also be helpful for transferring data. Hence, I will not talk about Gsutil and our prime focus is on AWS CLI. I have read several articles that shows a very simple approach of using commands to transfer data using Access Key ID, Secret Access Key, and region of the AWS S3 bucket. Refer article: cp — AWS CLI 1.19.5 Command Reference (amazon.com)

 

Problem

 

In spite of working with commands of AWS CLI. I was not able to access external the AWS S3 bucket. There are several UI tools available for connection with cloud environments like CloudBerry explorer, Cyberduck, etc. I tried using them also but I could not see AWS S3 bucket with Cloudberry but through cyberduck I was able to access the AWS S3 bucket but I cannot use this tool to transfer data because my data is not at my local system.- Now after explaining all my hurdles, I will move to the solution to this problem.

 

Prerequisites

 

To implement the functionality below are the prerequisites:

1. Install AWS CLI in the machine from where you want to transfer data. Below is the link for installing AWS CLI on Windows and Linux machines.
   
   For Windows - Installing, updating, and uninstalling the AWS CLI version 2 on Windows - AWS Command Line Interface (amazon.com)
   Linux - Installing, updating, and uninstalling the AWS CLI version 2 on Linux - AWS Command Line Interface (amazon.com)
   
   
2. Access Key ID, Secret Access Key, and region name of the AWS S3 bucket.

Commands

 

I had a Linux machine so I ran the below commands with the help of Putty. To transfer data, three steps are required, as shown below:

 

Configure AWS

 

[root@gcp6 testfolder]# sudo /usr/local/bin/aws configure

AWS Access Key ID [****************PYGA]: [AccessKeyID]

AWS Secret Access Key [****************Adly]: [SecretAccessKey]

Default region name [regionname]: [Your Region Name]

Default output format [JSON]: JSON

 

Test AWS S3 bucket connection

 

[sgupt@gcp6 ~]$ sudo /usr/local/bin/aws s3 ls [yourawsbucketname] --region [awsregionname]

 

Output

 

DummyFolder/

TestFolder/

 

Upload using cp command for specific files

 

[root@gcp03 testfolder]# sudo /usr/local/bin/aws s3 cp /home/sourcedirectorypath/test.txt s3://[yourawsbucketname]/[targetdirectoryname]/

 

Output

 

upload: ./test.txt to s3://[yourawsbucketname]/[targetdirectoryname]//test.txt

 

Upload using sync command for a complete directory

 

[root@gcp03 testfolder]# sudo /usr/local/bin/aws s3 sync /home/sourcedirectorypath /testfolder s3:// ://[yourawsbucketname]/[targetdirectoryname]/

 

Output

 

upload: ./test.txt to s3:// ://[yourawsbucketname]/[targetdirectoryname]/test.txt