Planning the Hardware & Cluster Resources

Sizing a Spark application's hardware allocation—specifically configuring executor counts, cores, and memory sizes—is one of the most critical steps in tuning a production big data pipeline. Incorrect configuration can lead to extreme performance bottlenecking, OutOfMemoryError (OOM) failures, or highly inefficient cloud execution bills.

This guide details the mathematical and architectural methodologies for sizing your Spark cluster resources on YARN.

1. The Core Bottlenecks of "Bad" Hardware Sizing

When sizing your Spark configurations, you must avoid three extreme resource states:

A. Tiny Executors (e.g., 1 Core, 1GB Memory)

• Problem: You lose the benefits of running multi-threaded processes. Sharing variables in memory via broadcast variables or shared caches becomes impossible because each executor runs only a single thread.
• Overhead: You pay high JVM launch and serialization overhead costs.

B. Giant Executors (e.g., 16 Cores, 64GB Memory)

• Problem: Garbage Collection (GC) pauses. If you assign more than 5–6 cores per executor, the garbage collector has to clean up massive JVM heaps, resulting in severe process stalls where the application freezes.
• HDFS Bottleneck: In YARN clusters, sharing more than 5 HDFS write streams per executor simultaneously can cause HDFS write throughput throttling.

C. Over-allocating Driver Memory

• Problem: Allocating too much memory to the Driver starves the executors. Driver memory should only be scaled up if you are using massive .collect() actions, big broadcast variables, or metadata tracking for millions of partitions.

2. The Recommended Formula: "5-Core Rule"

For optimal execution, production standards suggest using 5 cores per executor to balance multi-threading advantages with HDFS write constraints and GC pauses.

Let's walk through the math step-by-step to size a cluster:

Scenario Details:

• Physical Cluster Node Size: 6 Machines (Nodes), each having 16 CPU Cores and 64 GB RAM.

Step 1: Calculate Cores Available for Spark

On a Hadoop/YARN worker node, some hardware resources must be reserved for the operating system and YARN daemon processes (e.g., the NodeManager).

• Core Reserve: Reserve 1 Core per machine for OS / daemons.
• Available Cores per Node: 16 - 1 = 15 Cores
• Total Cluster Cores: 15 Cores × 6 Nodes = 90 Cores

Step 2: Determine Cores per Executor & Executor Count

Apply the optimal 5 cores per executor rule:

• Executors per Node: 15 Cores / 5 Cores = 3 Executors per Node
• Total Executors in Cluster: 3 Executors × 6 Nodes = 18 Executors

(Wait! We must reserve 1 executor slot in the cluster to run the Application Master/Driver if running in cluster mode. So total active Task Executors = 17 Executors).

Step 3: Calculate Memory per Executor

Now, calculate the memory allocation per node:

• Memory Reserve: Reserve about 10% of physical RAM for OS/YARN daemons.
• 10% of 64 GB = ~6.4 GB. Let's reserve 4 GB (leaving 60 GB for Spark per node).
• Total Available Spark Memory per Node: 60 GB
• Memory per Executor: 60 GB / 3 Executors per Node = 20 GB per Executor

Step 4: Subtract YARN Memory Overhead

YARN requires some additional container memory overhead to manage internal executor processing (e.g., off-heap memory, thread stacks).

• Overhead Formula: spark.yarn.executor.memoryOverhead defaults to 10% of executor memory or 384MB (whichever is larger).
• Overhead Value: 10% of 20 GB = 2 GB
• Final Executor Memory Allocation: 20 GB - 2 GB = 18 GB

3. Final Production Configuration Summary

Based on the calculations above, the optimized Spark submit configuration parameters for our 6-node cluster are:

spark-submit \
    --master yarn \
    --deploy-mode cluster \
    --num-executors 17 \
    --executor-cores 5 \
    --executor-memory 18g \
    --conf spark.yarn.executor.memoryOverhead=2048m \
    --driver-memory 18g \
    --driver-cores 5 \
    /path/to/script.py

4. Quick-Reference Rules of Thumb

1. Always use 4 to 5 cores per executor (--executor-cores 5).
2. Leave at least 1 Core and 4GB RAM per physical machine for the Operating System and Hadoop daemons.
3. Always account for YARN overhead (subtract 10% from your executor memory calculation).
4. If you run in cluster mode, subtract 1 executor from the total calculated cluster count to make room for the Application Master.