The Hadoop Ecosystem & HDFS Architecture

Master-Slave Cluster Architecture

A Hadoop cluster operates on a Master-Slave design pattern, cleanly decoupling storage metadata from compute execution:

flowchart TD
    subgraph Master["MASTER NODE"]
        A[NameNode<br/>Storage Metadata]
        B[ResourceManager / JobTracker<br/>Compute Master]
        style A fill:#fef08a,stroke:#eab308,stroke-width:2px,color:#000
        style B fill:#fef08a,stroke:#eab308,stroke-width:2px,color:#000
    end

    subgraph Slave1["SLAVE NODE 1"]
        C1[DataNode]
        T1[NodeManager / TaskTracker]
        style C1 fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#000
        style T1 fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#000
    end

    subgraph Slave2["SLAVE NODE 2"]
        C2[DataNode]
        T2[NodeManager / TaskTracker]
        style C2 fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#000
        style T2 fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#000
    end

    Master --- Slave1
    Master --- Slave2

1. The Storage Layer: HDFS

The Hadoop Distributed File System (HDFS) logically partitions massive files into large, uniform blocks (128MB default) and distributes them across worker nodes with a default 3x Replication Factor.

Core HDFS Components:

• NameNode (Master): Stores file namespace, directory tree, permissions, and block-to-node mappings in RAM for sub-millisecond lookup. Persists snapshot state to disk via FSImage and transaction logs via EditLog.
• Secondary NameNode: Periodically merges the active EditLog into FSImage (checkpointing) to prevent startup latency.
• DataNode (Worker): Stores physical data block files on local disks, sending heartbeats (every 3s) and block reports (every 6h) to the NameNode.

flowchart TD
    subgraph Rack1[Rack 1]
        A[Node A<br/>Replica 1]
        B[Node B]
        style A fill:#dcfce7,stroke:#22c55e,stroke-width:2px,color:#000
    end

    subgraph Rack2[Rack 2]
        C[Node C<br/>Replica 2]
        D[Node D<br/>Replica 3]
        style C fill:#dcfce7,stroke:#22c55e,stroke-width:2px,color:#000
        style D fill:#dcfce7,stroke:#22c55e,stroke-width:2px,color:#000
    end

    Rack1 <-- "Rack-Aware Replication" --> Rack2

HDFS Read & Write Pipelines

HDFS Read Sequence:

1. Client contacts NameNode to fetch sorted physical block locations (closest node first).
2. Client stream connects directly to the nearest DataNode via TCP sockets to read Block 1.
3. Stream sequentially switches to the next DataNode for subsequent blocks until completed.

HDFS Write Sequence:

1. Client requests NameNode to allocate a new block pipeline (DataNode 1 → DataNode 2 → DataNode 3).
2. Client streams data packets (64KB) to DataNode 1, which concurrently forwards packets down the pipeline to DataNode 2 and 3.
3. ACKs flow back upstream to verify block checksums before closing the pipeline lease.

High Availability & Advanced Storage

• Active/Standby HA: Eliminates Single Point of Failure (SPOF) using JournalNodes (quorum sync) and ZooKeeper (ZKFC) for automated failover and fencing.
• Erasure Coding (EC - Hadoop 3.x): Uses Reed-Solomon RS(6,3) striping to reduce storage overhead from 200% (3x Replication) down to 50%, ideal for cold data archives.

Ecosystem Tools Comparison