Introduction to Big Data Analytics Using Microsoft Azure

Introduction to Big Data

Big Data

Big Data refers to data that is too large or complex for analysis in traditional databases because of factors such as the volume, variety, and velocity of the data to be analyzed.

Volume

Volume is the quantity of data that is generated. For example, consider analyzing application logs, where new data is generated each time a user does some action in an application. This may generate several lines per minute or even per second as the user works.

Variety

The data that needs to be analyzed is not standard, consisting of both structured and unstructured data. One example of this can be the analysis of Social Media data consisting of emoticons, hashtags and texts in several languages.

Velocity

This is where data is being generated very frequently. This is becoming quite common with emerging technologies such as the Internet of Things where devices/sensors generate data continuously.

Apache Hadoop

Apache Hadoop is an open-source Java framework primarily intended for storage and processing of very large sets of data. It does distribute processing of large data sets where the data is split across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

MapReduce

MapReduce is the application logic that splits the data for processing by various nodes in the Hadoop cluster. A MapReduce job usually splits the input data-set into independent chunks that are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps that are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file system. The framework also takes care of scheduling tasks, monitoring them, and re-executing the failed tasks. MapReduce is done in the following 3 steps:

Introduction to Azure HDInsight

Azure HDInsight deploys and provisions Apache Hadoop clusters in the cloud, providing a software framework designed to manage, analyze, and report on big data. The Hadoop core provides reliable data storage with the Hadoop Distributed File System (HDFS), and a simple MapReduce programming model to process and analyze, in parallel, the data stored in this distributed system.

Creating an HDInsight Cluster

To create an Azure HDInsight Cluster, open the Azure portal then click on New > Data Services > HDInsight. The following options are available:

This article uses the Hadoop cluster. The next step is to add a cluster name, select the cluster size, add a password, select storage, and click on create HDInsight cluster.

Enable Remote Desktop on the Cluster

Once the cluster has been created, its jobs and contents can be viewed by remote connection. To enable remote connection to the cluster, use the following procedure:

In the Configure Remote Desktop wizard, enter a user name and password for the remote desktop. Note that the user name must be different from the one used to create the cluster (admin by default with the Quick Create option). Enter an expiration date in the EXPIRES ON box.

Accessing the Hadoop Cluster using Remote Desktop Connection

To connect to the cluster via Remote Desktop Connection, in the portal, select your cluster and go to configuration and click connect. An RDP file will be downloaded that shall be used to connect to the cluster. Open the file, enter the required credentials and click connect. Once the Remote Connection is established, double-click the Hadoop Command Line icon. This will be used to navigate through the Hadoop File System.

View files in the root directory

Once the command line is open, you may view all the files in the root folder. The syntax to use is Hadoop fs followed by the Linux command used inside the Hadoop File System.