Theoretical Quiz: spark-submit Fundamentals
Test your conceptual understanding of how Spark applications are deployed and managed across a cluster.
Q1: Client vs. Cluster Deploy Mode
Explain the architectural difference between --deploy-mode client and --deploy-mode cluster. Where does the Driver process live in each scenario? Which mode is recommended for interactive Notebooks (like Jupyter), and which is recommended for scheduled production ETL pipelines?
Q2: The Role of the Cluster Manager
Spark supports several cluster managers (YARN, Mesos, Kubernetes, Standalone). What is the exact role of the Cluster Manager during a spark-submit? Does the Cluster Manager actually execute your Python code?
Q3: spark-defaults.conf
You have set spark.executor.memory=2g in the spark-defaults.conf file on your edge node. However, when running spark-submit, you explicitly pass the flag --executor-memory 4g. Which memory setting will Spark actually use, and why? What is the order of precedence for Spark configurations?
Q4: Dependency Distribution
In PySpark, if your application requires a third-party Python library (e.g., pandas or requests) that is not installed on the cluster nodes, how can you use spark-submit to distribute a virtual environment (like a .tar.gz or .pex file) to all executors?
Q5: Exit Codes
If the Driver process crashes in cluster mode on YARN, what exit code will spark-submit return to the scheduling tool (like Airflow) that triggered it? How does the scheduler know the job failed?