Hands-on Quiz: Mastering spark-submit
This practical quiz will test your ability to package and deploy a PySpark application using spark-submit.
Scenario Setup
Imagine you have a project directory with the following structure:
my_spark_project/
├── main.py
├── utils/
│ ├── __init__.py
│ └── text_helpers.py
└── configs/
└── job_config.json
Inside main.py, you are importing from utils.text_helpers import clean_text and reading the configuration from configs/job_config.json.
You need to run this job on a Hadoop YARN cluster.
Questions
Q1: Deploy Mode
You want to run this job in a production YARN cluster. You do not want the job to fail if your SSH connection to the edge node drops. Which --deploy-mode should you use?
Q2: Managing Dependencies (Python modules)
If you just run spark-submit --master yarn main.py, your executors will throw a ModuleNotFoundError: No module named 'utils'.
What command-line steps must you take to correctly package the utils directory and pass it to spark-submit so the executors can find it?
Q3: Passing Data/Config Files
Your script also needs to read configs/job_config.json. On a distributed cluster, the executors won't have this file on their local disk.
Which spark-submit flag should you use to distribute this JSON file to the working directory of every executor?
Q4: Dynamic Allocation vs. Static Allocation
Write the full spark-submit command to deploy main.py to YARN in cluster mode. Explicitly request 4 executors, with 2 cores and 4GB of memory per executor.
Q5: Debugging Failures
Your job failed in YARN cluster mode. You check the console where you ran spark-submit, but there are no Python tracebacks, only a message saying State: FAILED. What CLI command do you run (using the YARN application ID) to fetch the actual driver logs and see the Python error?
Hint: Look into the --py-files and --files arguments for Q2 and Q3!