Spark-Submit Utility

spark-submit is a command-line utility used to launch and run Apache Spark applications on a cluster. It packages your application code (written in Python, Scala, Java, or R) and submits it to a cluster manager (such as YARN, Kubernetes, Mesos, or Standalone) for distributed execution.

Spark-Submit Execution Flow

When you run spark-submit, the Spark environment initiates the following application launch process:

1. Client Request: The client runs spark-submit with your job parameters and application file.
2. Resource Allocation: spark-submit requests resource allocation from the active Cluster Manager (e.g., YARN Resource Manager).
3. Application Master & Driver: The Cluster Manager provisions a container to host the Application Master and start the Spark Driver (containing the DAGScheduler and TaskScheduler).
4. Executor Workers: The Driver coordinates with the Cluster Manager to spawn multiple Spark Executor worker processes across the cluster's compute nodes to compute partition tasks.

Why is it used?

In a typical development workflow, you might write and test Spark code interactively using Jupyter Notebooks, Zeppelin, or PySpark shell. However, for production pipelines and automated batch jobs, interactive shells are not viable.

spark-submit is the bridge that transitions development code into highly scalable production pipelines because:

• It supports all major Cluster Managers (Local, YARN, Kubernetes, Standalone).
• It handles complex dependency distribution (transferring external .jar, .py, or .zip files to all workers).
• It abstracts the underlying deployment modes, meaning the exact same command template works whether you scale on a local development laptop or run across thousands of machines.

Core Command Syntax & Arguments

The basic syntax of a spark-submit command is:

spark-submit [options] <application-file> [application-arguments]

1. Essential Cluster Arguments

• --master: Specifies the cluster manager connection URL.
• local[N]: Run locally with N worker threads (usually matching your machine's CPU cores).
• yarn: Submit to a Hadoop YARN cluster.
• k8s://https://<api_endpoint>: Submit to a Kubernetes cluster.
• spark://<host>:<port>: Submit to a Spark Standalone cluster.
• --deploy-mode: Determines where the Spark Driver runs.
• client: The Driver runs locally on the machine where the spark-submit command is triggered. Useful for debugging and interactive work.
• cluster: The Driver is launched as a container inside the cluster by the cluster manager. Ideal for production batch pipelines to avoid network latency.

2. Resource Allocation Arguments

• --num-executors: Total number of executor processes to launch across the cluster (YARN/K8s only).
• --executor-cores: Number of CPU cores allocated to each executor. This directly controls the number of concurrent tasks an executor can run.
• --executor-memory: Total heap memory allocated to each executor process (e.g., 8g, 16g).
• --driver-memory: Total memory allocated for the Spark Driver process (e.g., 4g).
• --driver-cores: CPU cores to allocate to the Driver (cluster mode only).

3. Application Packaging & Dependency Arguments

• --class: The entry point class name (for compiled Java/Scala applications only, e.g. com.example.ETLJob).
• --jars: A comma-separated list of local/S3 paths to external .jar packages needed by your job (e.g., database JDBC drivers).
• --py-files: A comma-separated list of .py, .zip, or .egg files to add to the Python path for PySpark jobs (ideal for distributing custom helper modules).
• --files: Comma-separated list of static files (JSON, CSV, properties) to upload to each executor's local working directory.

Simple Execution Template

Here is a standard example of submitting a python ETL script to a local machine:

spark-submit \
    --master "local[4]" \
    --name "Local-Batch-ETL" \
    --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \
    /path/to/my_etl_job.py \
    --input_path "/data/raw/" \
    --output_path "/data/processed/"