Big Data Architecture: Theoretical System Design
Series
Data Engineering & Distributed Systems Series
Estimated Time
~35 Mins Quiz
Quiz Overview
FAANG-style senior system design scenarios covering HDFS pipeline failures, data skew stragglers, Active/Standby fencing, and 3x Replication vs Erasure Coding.
Scenario 1: HDFS Write Pipeline Node Failure
Problem: During an active block write (DataNode 1 → DataNode 2 → DataNode 3), DataNode 2 crashes with buffered packets in ackQueue.
Resolution Flow:
- Pipeline Freeze & Re-queue:
DFSOutputStreamhalts transmission and shifts all unacknowledged packets from theackQueueback into the front of thedataQueue. - Re-register Pipeline: The client contacts the NameNode, which removes the crashed
DataNode 2and allocates a clean pipeline (DataNode 1 → DataNode 3) with a new Generation Stamp. - Self-Healing Replication: After the writing pipeline completes, the NameNode detects that the block is under-replicated (only 2 physical copies exist) and asynchronously streams a third copy to a new worker.
Scenario 2: Data Skew & Reducer Stragglers
Problem: One viral tracking key represents 90% of a 10TB dataset, causing a single Reducer task to hang at 99% for hours. Speculative execution fails to solve the lag.
Resolution Flow:
- Why Speculative Fails: Speculative execution only resolves hardware-induced stragglers. Because the lag is caused by sheer data volume (9TB skew), backup tasks spawned will run just as slowly.
- Fix (Key Salting): Append random hash suffixes (e.g.
"hot_key_1","hot_key_2") during the Map phase to distribute the 9TB uniformly across 50 intermediate Reducers, followed by a lightweight 2nd-stage merge job.
Scenario 3: Active/Standby Fencing & Split-Brain
Problem: An Active NameNode freezes during a long JVM GC pause. The Standby attempts auto-failover, resulting in both NameNodes trying to write edits, corrupting cluster metadata.
Resolution Flow:
- QJM Epoch Fencing: The Standby receives a higher epoch number (e.g.,
Epoch 6). JournalNodes automatically reject any subsequent write requests from the old Active node (Epoch 5). - Active Fencing (sshfence / powerfence): ZooKeeper Failover Controller (ZKFC) logs in via SSH to forcefully terminate the frozen master (
kill -9) or sends a signal via Power Distribution Units (PDU) to cut server rack power before promoting the Standby.
Scenario 4: 3x Replication vs Erasure Coding
Find this content helpful?
☕ Buy me a coffee