Step By Step Installation Of Pseudo Distribution Hadoop Cluster

The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation.

The uri's scheme determines the config property fs.SCHEME.impl) naming the FileSystem implementation class.

The uri's authority is used to determine the host, port, etc. for a filesystem.

Modifying hdfs-site.xml

The hdfs-site.xml file contains the configuration settings for HDFS daemons.

• NameNode
• Secondary NameNode
• DataNodes.
• CheckPoint - To Find the data Node is active or Dead. (During the specified time intervals DN send the information to the NameNode & NameNode will Monitor this, If there is no such transfer of signals happen from DataNode to Name Node, The NameNode will announce the DataNode as Dead for this purpose the checkpoint is being used)

In the hdfs-site.xml we need to specify default block replication (No., of Replicas for the Data Nodes).

The actual number of replications can also be specified when the file is created. The default is used if replication is not specified in create time.

In my cluster, I have set replicas as 1 only it can be modified based on the needs or requirements. Normally the replicas should be more than 1 in number. In-Order to avoid the Data Loss.

• sudo vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  
• Modifying the hdfs-site.xml using Vi Editor, add the contents which are present only between the configuration tags <configuration> </configuration>
  
  1. <configuration>
  2. <property>
  3. <name>dfs.replication</name>
  4. <value>1</value>
  5. <description>Default block replication. The actual number of replications can be specified when the file is created.The default is used if replication is not specified in create time. </description>
  6. </property>
  7. <property>
  8. <name>dfs.namenode.name.dir</name>
  9. <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
  10. </property>
  11. <property>
  12. <name>dfs.datanode.data.dir</name>
  13. <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
  14. </property>
  15. <property>
  16. <name>dfs.namenode.checkpoint.dir</name>
  17. <value>file:/usr/local/hadoop_store/hdfs/secondarynamenode</value>
  18. </property>
  19. <property>
  20. <name>dfs.namenode.checkpoint.period</name>
  21. <value>3600</value>
  22. </property>
  23. </configuration>

Modifying the yarn-site.xml file:

• sudo vi /usr/local/hadoop/etc/hadoop/yarn-site.xml
  
• The yarn-site.xml file contains configuration information that overrides the default values for YARN parameters.
  
  1. <configuration>
  2. <!-- Site specific YARN configuration properties -->
  3. <property>
  4. <name>yarn.nodemanager.aux-services</name>
  5. <value>mapreduce_shuffle</value>
  6. </property>
  7. <property>
  8. <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  9. <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  10. </property>
  11. </configuration>
  

Running All The Process

It will be a difference based on the Node Cluster which we have configured & Installed.

Single Node Cluster

• start-all.sh

• start-yarn.sh (Resource Manager and Node manager)
• start-dfs.sh (namenode, datanode and secondarynamenode)