Hands-on Quiz: Exploring the Spark UI

This hands-on quiz is designed to test your practical ability to navigate and extract insights from the Spark Web UI.

Scenario Setup

To complete this quiz, you will need a running Spark application. Start a PySpark shell or a local Spark session and execute the following code:

# Create a dummy DataFrame
df = spark.range(0, 10000000).withColumnRenamed("id", "value")

# Perform some transformations
df_filtered = df.filter("value % 2 == 0")
df_grouped = df_filtered.groupBy(df_filtered.value % 10).count()

# Trigger an action
df_grouped.collect()

Questions

Q1: Accessing the UI What port did your Spark UI start on? If 4040 was taken, where can you check the console logs to find the exact URL of the Spark UI?

Q2: Inspecting Jobs Navigate to the "Jobs" tab. How many jobs were triggered by the collect() action above? Why?

Q3: Analyzing Stages Click into the Job generated by your collect() action.

1. How many stages does this job have?
2. What operation caused the boundary between these stages? (Hint: Look for the word "Exchange").

Q4: Task Distribution Navigate to the "Stages" tab and look at the stage that performed the count().

1. How many tasks were executed in this stage?
2. Why were that specific number of tasks created? (Hint: Think about default shuffle partitions).

Q5: SQL Physical Plan Navigate to the "SQL" tab and click on the query corresponding to your action.

1. What type of physical aggregate operator was used? (e.g., HashAggregate, SortAggregate)
2. Follow the DAG from bottom to top. Can you identify where the filter operation took place?

Q6: Storage In your code, add the line df_filtered.cache() right after it is defined, and trigger an action (like df_filtered.count()). Now navigate to the "Storage" tab.

1. What is the storage level of the cached RDD?
2. How much memory did it consume?

Take your time exploring the UI to answer these questions! Getting comfortable with this interface is essential for debugging real-world Spark applications.