Scenario: YARN Production ETL

Scenario: Daily Enterprise Transaction Processing

The Challenge

A data engineering pipeline needs to process 500 GB of daily transactional logs stored in a Hadoop cluster's HDFS. The job parses raw logs, joins them with customer master records, calculates daily statistics, and writes the output back as parquet. The pipeline must execute under YARN cluster resource constraints on a corporate multi-tenant cluster.

1. Optimal Spark-Submit Configuration

spark-submit \
    --master yarn \
    --deploy-mode cluster \
    --name "Enterprise-Transaction-ETL" \
    --num-executors 20 \
    --executor-cores 5 \
    --executor-memory 16g \
    --driver-memory 8g \
    --driver-cores 2 \
    --conf spark.yarn.executor.memoryOverhead=2048m \
    --conf spark.sql.shuffle.partitions=400 \
    --conf spark.dynamicAllocation.enabled=true \
    --conf spark.dynamicAllocation.minExecutors=5 \
    --conf spark.dynamicAllocation.maxExecutors=40 \
    --jars hdfs:///libs/gcs-connector-hadoop3.jar \
    hdfs:///scripts/transaction_etl.py \
    --input_date "2026-05-31"

2. Parameter Explanations & Rationale

• --master yarn: Directs Spark to schedule resource containers using Hadoop's YARN ResourceManager.
• --deploy-mode cluster: Launches the Spark Driver inside a YARN container on the cluster. This isolates the submitting node from network disruptions and ensures that driver logs are managed by YARN.
• --num-executors 20 / --executor-cores 5: Provisions 100 total concurrent compute slots across the cluster. 5 cores per executor is chosen to balance HDFS write limits and avoid GC pauses.
• --executor-memory 16g: Provides sufficient heap memory to process standard HDFS blocks and support local caching.
• --conf spark.sql.shuffle.partitions=400: Sets partition size during shuffle transformations to 400. With 500GB of input data, this yields roughly ~1.25 GB partitions per task. Coupled with Spark's execution engine, this ensures tasks don't spill to disk.
• --conf spark.dynamicAllocation.enabled=true: Enables YARN to de-allocate executors if the job goes through low-compute phases (e.g., waiting for writes), releasing resources to other users on a shared cluster.