Spark Architecture: Theoretical System Design
Series
Data Engineering & Distributed Systems Series
Estimated Time
~30 Mins Quiz
Quiz Overview
FAANG-style senior Spark engineering scenarios covering Driver OOM collect crashes, lineage fault recovery, reduceByKey vs groupByKey shuffle boundaries, and Client vs Cluster deployment modes.
Scenario 1: Driver OutOfMemory (OOM) on `.collect()`
Problem: A PySpark job filters a 15TB dataset down to 45GB. Calling .collect() instantly crashes the Driver with java.lang.OutOfMemoryError: Java heap space.
Solution:
- JVM Mechanics:
.collect()pulls all distributed partitions across the network into the single Driver JVM process. Since 45GB exceedsspark.driver.memory(e.g. 4GB), the Driver heap overflows and crashes. - Safe Alternatives:
- Preview records with
.take(100)or.first(). - Write directly to distributed storage via
.saveAsTextFile("hdfs://..."). - Process elements inside worker executors using
.foreach().
Scenario 2: Lineage Fault Recovery & Wide Dependencies
Problem: Executor 4 crashes during an active RDD write. How does Spark recover Partition 3?
Solution:
- Narrow Recovery: For narrow dependencies (
map,filter), Spark consults the RDD lineage graph and recomputes only the single lost Partition 3 on a healthy worker node from the parent partition. - Wide Shuffle Boundaries: For wide dependencies (
reduceByKey), data is shuffled across nodes and written to shuffle files. Losing a partition post-shuffle requires re-executing all tasks from the entire stage preceding the shuffle.
Scenario 3: `reduceByKey` vs `groupByKey`
Problem: Auditing legacy PySpark code comparing groupByKey().mapValues(sum) vs reduceByKey(a + b).
Solution:
- Map-Side Combine:
reduceByKeyperforms local pre-aggregation inside mapper buffers before network shuffle.groupByKeysends all raw key-value pairs over the network. - Executor OOM Risk: On skewed datasets,
groupByKeyforces millions of raw values into a single executor's memory list, causing frequent JVM heap crashes.
Scenario 4: Deployment Modes (Client vs Cluster)
Find this content helpful?
☕ Buy me a coffee