Introduction To Hadoop Framework

Introduction

This blog is to teach you about Hadoop Framework.

Hadoop

  • Hadoop is an open-source framework.
  • Hadoop Framework is used to sort our application data (large amount of data).
  • Hadoop Framework is used to store any kind of data, as it has a large amount of storage space.
  • Hadoop Framework is used to handle data virtually.

Hadoop History

  • 1990 Apache Software foundation introduced Hadoop Framework technology.
  • 2004 Google Publishes GFS paper.
  • 2005 Nutch uses MapReduce.
  • 2008 Becomes Apache top-level project.
  • 2009 Yahoo uses hadoop.
  • 2013 Hadoop is used in many companies.

We need to understand Big Data before learning about the Hadoop Framework.

Big Data

  • Big Data is a collection of large amounts of data that cannot be processed by our computer technologies. Big Data technologies provide a very accurate analysis, so it's useful to know the results.
  • Big data has some challenges. It helps in capturing the data, curation, storage, searching, sharing, transfer, analysis and presentation.

Hadoop is important

  • Quickly store & process large amount of data.
  • Computing power.
  • Fault tolerance.
  • Flexibility & low cost.
  • Scalability.

Challenges in Hadoop

  • MapReduce programming is not matched for all problems.
  • Data security issues.
  • Hadoop is not easy to use.

Hadoop Data Gathering

Here, we will learn how to add our data to Hadoop.

  • Third-party vendor connectors (SAS/ACCESS or SAS Data Loader for Hadoop) are used to update our data to Hadoop
  • Apache Flume is used to load the data to Hadoop.
  • Some simple Java commands are used to transfer the data form the files into Hadoop.

Hadoop Components

  • HDFS-Distributed System.
  • MapReduce-Distributed Data Processing Model.

Hadoop Architecture

  • It operates on top of an existing file system.
  • Its files are stored as the blocks.
  • It provides reliability through replication.
  • NameNode stores the metadata and manages access.
  • No data caching due to large datasets.

Next Recommended Reading Why Hadoop Is Needed For Big-Data