Introduction
As we saw in the prior article, every machine works on its own portion of data. Let's talk about data storage strategies and key design goals/assumptions.
HDFS Design Goal
- Reliability
- We should not lose data in any scenario.
- We use many hardware devices and inevitably something will fail (Hard Disk, Network Cards, Server Rack, and so on) at some point or another.
- Scalability
- We should be able to add more hardware (workers) to get the job done.
- We should be able to add more hardware (workers) to get the job done.
- Cost-Effective
- The system needs to be cheap, after all, we are building a poor man's supercomputer and we do not have a budget for fancy hardware.
Data Assumptions
- Process large files both horizontally and Vertically (GigaByte +).
- Data is append-only.
- Access to data is large and sequential.
- Write Once and Read many times.
Throughput vs. Random Seek
Since we are working with large datasets with sequential read, Hadoop and Map Reduce is optimized for throughput and not random seek.
In other words, if one must move a big house from one city to another than a slow big truck will do a better job, than a fancy small fast car.
Data Locality
It is cheaper to move the compute logic than data. In Hadoop, we move the computation code around where the data is present, instead of moving the data back and forth to the compute server; that typically happens on a client/server model.
Data Storage in HDFS
Let's say we need to move a 1 Gig text file to HDFS.
Join the conversation! Your thoughts help the community grow.