Theoretical Quiz: Understanding the Spark UI
Test your theoretical knowledge of the Spark Web UI and its components.
Q1: Job vs. Stage What is the fundamental difference between a Spark "Job" and a Spark "Stage" as represented in the Web UI? What triggers the creation of a new Job, and what causes a Job to be split into multiple Stages?
Q2: The Role of Port 4040
By default, the Spark Web UI starts on port 4040. If you launch a second Spark application on the same machine while the first is still running, what will happen to the Web UI for the second application?
Q3: Diagnosing Data Skew If you suspect your application is suffering from data skew, which specific tab in the Spark UI would you navigate to, and what specific metric or visual indicator would confirm your suspicion?
Q4: Garbage Collection (GC) Overhead You notice in the "Executors" tab that one of your executors has a very high "Task Time (GC Time)". For example, tasks are taking 10 seconds, but 4 seconds of that is spent in GC. What does this indicate about your executor's memory, and what are two ways you could address this issue?
Q5: The SQL Tab When running PySpark DataFrame code, the "SQL" tab becomes available. This tab visualizes the execution plan. Which component of the Spark engine is responsible for generating this physical plan shown in the UI?
Q6: Dead Executors If an executor runs out of memory and crashes (OOM), how does this appear in the Spark UI? Where would you look to find the exact reason for the executor's failure?
Q7: Storage Level
In the "Storage" tab, what does the storage level Memory Deserialized 1x mean? How does it differ from Disk Serialized 1x?
Take time to think through these answers. They represent common interview questions and real-world scenarios you will face as a Data Engineer.