Spark Cluster Architecture & Fundamentals

1. Scala vs. PySpark Architecture

graph TD
    subgraph ScalaEngine["Spark with Scala (Native)"]
        SC["Scala Code"] -->|Direct Execution| JVM["Spark Core JVM Engine"]
    end

    subgraph PythonEngine["Spark with Python (PySpark)"]
        PY["Python Code"] -->|Py4J Gateway| Bridge["Socket JVM Bridge"]
        Bridge --> JVM
        JVM -->|Data Serialization| PY_Worker["Python Worker Processes"]
    end

    style ScalaEngine fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
    style PythonEngine fill:#fff3e0,stroke:#e65100,stroke-width:2px;

• Structured APIs (DataFrames & SQL): Performance is identical. Catalyst compiles Python or Scala code into identical native JVM bytecode.
• Low-level RDD APIs: Scala is faster because PySpark must serialize data via Py4J and Pickling across socket bridges to local Python worker processes.

2. Driver vs. Executor Architecture

flowchart TD
    subgraph Master["DRIVER NODE (Master Process)"]
        D1[Driver Program]
        D2[SparkSession & DAG Scheduler]
        D1 --- D2
    end

    subgraph CM["CLUSTER MANAGER"]
        YARN[YARN / K8s / Standalone]
    end

    subgraph Workers["EXECUTOR WORKER NODES"]
        E1[Executor 1 JVM<br/>Task Pool]
        E2[Executor 2 JVM<br/>Task Pool]
    end

    Master <--> CM <--> Workers

1. Driver: Coordinates execution, maintains SparkSession, compiles execution plans into physical DAG stages, and schedules tasks to executors.
2. Cluster Manager: Allocates cluster resources across YARN, Kubernetes, or Standalone managers.
3. Executors: Worker JVM processes executing individual task partitions and storing in-memory blocks.

3. Initializing a PySpark Session

The SparkSession is the single unified entry point for DataFrames, SQL, and cluster configurations:

from pyspark.sql import SparkSession

# Initialize a SparkSession
spark = SparkSession.builder \
    .appName("PySpark Architecture Intro") \
    .getOrCreate()

# Create DataFrame
data = [("Alice", 28), ("Bob", 32), ("Charlie", 25)]
df = spark.createDataFrame(data, ["Name", "Age"])
df.show()