Spark Version Evolution
"Trace Apache Spark's chronological evolution — from RDD genesis at UC Berkeley to DataFrames, Catalyst, Tungsten, Adaptive Query Execution (AQE), and Spark Connect 4.0."
Why Spark Replaced MapReduce
In 2009, UC Berkeley AMPLab researchers identified MapReduce's core flaw: disk-bound iterative execution. Iterative algorithms (Machine Learning, Graph Processing, PageRank) spend up to 90% of execution time writing intermediate states back to HDFS disks.
Spark introduced Resilient Distributed Datasets (RDDs):
- Maintains intermediate processing states in Executor RAM.
- Tracks a logical DAG lineage graph to reconstruct lost partitions automatically.
- Runs iterative workloads up to 100x faster than Hadoop MapReduce.
Spark Evolutionary Eras
Spark 1.x (2014 - 2016): The Structured Era
Introduced RDDs, Spark SQL, the DataFrame API (Catalyst Optimizer), and the Dataset API for Scala/Java.
Spark 2.x (2016 - 2019): Tungsten & Streaming
Introduced the Tungsten Engine (whole-stage code generation, off-heap memory management) and Structured Streaming.
Spark 3.x (2020 - 2024): Intelligent Optimization
Added Adaptive Query Execution (AQE) (dynamic partition coalescing, skew join handling), Dynamic Partition Pruning (DPP), and native Pandas API integration.
Spark 4.0 (2025+): Distributed Client Architecture
Features Spark Connect (gRPC thin client connecting to remote clusters without heavy drivers), PySpark Arrow-vectorized execution, and RocksDB state stores.