Brief Description Of HDFS Architecture

HDFS Concepts

Name Node

• Manages the process of replication and distribution.
• Keeps track of block location and complete file system.
• Smart replication saves system from failure and provides resilience.
• Provides the data from replicas of name node if any of the node or rack fails.

Data Node

• Physically stores the data in files.

HDFS Ideas

• Handles very large file.
• 100s of MB, GB, TB and above.
• Data Access.
• Writes once and reads many times.
• Commodity hardware.
• Price ranged between $1,000 to $5,000. Not like in RDBMS $50,000.

HDFS is not designed to,

• Low-latency of data access (cannot read data quickly).
• Doesn’t fit OLTD database (needs RDBMS).
• Small files.
• More the number of small files, more the memory of name node be taken up.
• Arbitrary file modifications.
• Only appends are supported.

HDFS Blocks

• Block size is the minimum size of the data that can be read or written to a file system.
• Typically, for a disk, it is 512 bytes, but for HDFS, its default is 64 MB.
• A file smaller than a block size doesn’t occupy a full block size in HDFS.

Why the block size is so large

• Seek time is around 10ms.
• At transfer rate 100mbps, seek time is around 1% of the transfer time.
  
  
  HDFS Architecture

Fig. HDFS Architecture

• HDFS is meant for very large files and won’t work efficiently with lots of small files.
  
  
• Write once, read many times is the principle. However, appends are supported.
  
  
• HDFS works on master-slave architecture. The name node is the master and Data nodes are workers/ slave nodes.
  
  
• Name node maintains namespace images and edit-log in the main memory.
  
  
• Failure of name node leads to a complete system failure called Single Point Of Failure (SPOF). Hence, its hardware has to be more resilient. Still, with updates, the hardware failures happen, so to counter those situations, resilience additions are,
• Remote NFS mount.
• Secondary Name Node.
  
  
• Secondary name node timely combines the edit log and namespace so as to avoid the name node to run out of memory.
  
  
• Specification of secondary name node should be equivalent to the primary name node.
  
  
• Name node is responsible for timely updating the namespace image and editing the log to local disk and on a remote network storage location.
  
  
• In case the failure of the last information from NFS mount is retrieved manually, the administrator to machine (secondary name node) configures it as name node. It takes approx. 30 minutes to get stable.