RDD Foundations & Architecture

What is an RDD?

An RDD is an immutable, partitioned collection of records operated on in parallel across a cluster of worker nodes:

• Resilient: Self-healing via lineage graphs — lost partitions are recomputed automatically without data replication.
• Distributed: Dataset is divided into logical partitions processed in parallel across cluster nodes.
• Dataset: Read-only collection of typed objects (tuples, rows, key-value pairs).

The 5 Core RDD Properties

classDiagram
    class RDD {
        +List Partitions
        +List Dependencies
        +Function Compute
        +Partitioner partitioner
        +List PreferredLocations
    }

1. Partitions List: The physical units of parallelism.
2. Dependency List (Lineage): Tracks parent RDDs to rebuild lost partitions.
3. Compute Function: Applies transformation logic to an iterator of partition records.
4. Partitioner (Optional): Hashes keys across worker nodes (e.g. HashPartitioner).
5. Preferred Locations (Optional): Enforces Data Locality by executing tasks on nodes hosting the underlying HDFS blocks.

Lazy Evaluation & Lineage Fault Recovery

Spark delays execution of Transformations (map, filter, flatMap) until an Action (collect, count, saveAsTextFile) is called. When triggered, the DAG Scheduler compiles the lineage graph into physical stages and parallel tasks.

graph LR
    Input[("HDFS File")] -->|textFile| RDD1["RDD 1 (Lines)"]
    RDD1 -->|filter| RDD2["RDD 2 (Errors)"]
    RDD2 -->|map| RDD3["RDD 3 (Messages)"]
    RDD3 -->|collect| Output["Driver Program"]

    style RDD1 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
    style RDD2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
    style RDD3 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;

If a worker node crashes mid-job, Spark uses the RDD's lineage graph to recompute only the missing partition on a healthy node, avoiding heavy 3x disk replication.

MapReduce vs RDD Comparison