Introduction
In one of my projects, I had the requirement to transfer GBs of data from one cloud platform to another cloud platform (i.e. Google Cloud Platform to Amazon Web Service S3 bucket). I had a Linux virtual machine where my data is kept in a certain directory and I have to transfer that data to some external client that has an AWS S3 bucket.
Options to implement the above scenario
Gsutil
We don’t want to have the extra baggage of creating a GCP bucket so I decided not to work for this approach and I had to look for other options which are shown below.
AWS CLI
Now after evaluating my first approach which as per my requirement is not the correct way. I found AWS CLI will also be helpful for transferring data. Hence, I will not talk about Gsutil and our prime focus is on AWS CLI. I have read several articles that shows a very simple approach of using commands to transfer data using Access Key ID, Secret Access Key, and region of the AWS S3 bucket. Refer article:
cp — AWS CLI 1.19.5 Command Reference (amazon.com)
Problem
In spite of working with commands of AWS CLI. I was not able to access external the AWS S3 bucket. There are several UI tools available for connection with cloud environments like CloudBerry explorer, Cyberduck, etc. I tried using them also but I could not see AWS S3 bucket with Cloudberry but through cyberduck I was able to access the AWS S3 bucket but I cannot use this tool to transfer data because my data is not at my local system.- Now after explaining all my hurdles, I will move to the solution to this problem.
Prerequisites
To implement the functionality below are the prerequisites:
Commands
I had a Linux machine so I ran the below commands with the help of Putty. To transfer data, three steps are required, as shown below:
Configure AWS
[root@gcp6 testfolder]# sudo /usr/local/bin/aws configure
AWS Access Key ID [****************PYGA]: [AccessKeyID]
AWS Secret Access Key [****************Adly]: [SecretAccessKey]
Default region name [regionname]: [Your Region Name]
Default output format [JSON]: JSON
Test AWS S3 bucket connection
[sgupt@gcp6 ~]$ sudo /usr/local/bin/aws s3 ls [yourawsbucketname] --region [awsregionname]
Output
DummyFolder/
TestFolder/
Upload using cp command for specific files
[root@gcp03 testfolder]# sudo /usr/local/bin/aws s3 cp /home/sourcedirectorypath/test.txt s3://[yourawsbucketname]/[targetdirectoryname]/
Output
upload: ./test.txt to s3://[yourawsbucketname]/[targetdirectoryname]//test.txt
Upload using sync command for a complete directory
[root@gcp03 testfolder]# sudo /usr/local/bin/aws s3 sync /home/sourcedirectorypath /testfolder s3:// ://[yourawsbucketname]/[targetdirectoryname]/
Output
upload: ./test.txt to s3:// ://[yourawsbucketname]/[targetdirectoryname]/test.txt