PySpark RDD Execution Tracing: Clickstream Logs
Series
Data Engineering & Distributed Systems Series
Estimated Time
~35 Mins Lab
Lab Objective
Trace user clickstream log records step-by-step through PySpark RDD narrow transformations (`filter`, `map`), wide shuffles (`reduceByKey`), and driver action collections (`collect`).
Input Clickstream Logs & Setup
Dataset:
user1,click,home,2026-05-25T10:00:00
user2,view,product_page,2026-05-25T10:01:00
user1,click,checkout,2026-05-25T10:02:00
user3,view,home,2026-05-25T10:03:00
user1,click,payment_gateway,2026-05-25T10:04:00
PySpark Pipeline Code:
raw_rdd = sc.textFile("clickstream.txt", minPartitions=2)
clicks_rdd = raw_rdd.map(lambda line: line.split(",")) \
.filter(lambda cols: cols[1] == "click") \
.map(lambda cols: (cols[0], 1))
result_rdd = clicks_rdd.reduceByKey(lambda a, b: a + b)
output = result_rdd.collect()
Execution Trace Solution
1. Narrow Transformation Trace (map & filter):
- Partition A (Worker 1 - Rows 1-2): Emits
[("user1", 1)](Row 2 view event filtered out). - Partition B (Worker 2 - Rows 3-5): Emits
[("user1", 1), ("user1", 1)](Row 4 view event filtered out).
2. Wide Transformation Shuffle (reduceByKey):
- Partitioner formula:
Partition = username_last_digit % 2. user1(1 % 2 = 1) routes to Partition 1 (Worker 2).- Worker 1 sends
("user1", 1)over TCP to Worker 2. - Worker 2 retains its local tuples
[("user1", 1), ("user1", 1)]locally without network transfer.
3. Aggregation & Driver Action (collect):
- Worker 2 (Partition 1) aggregates:
1 + 1 + 1 = 3Produces("user1", 3). - Driver Node calls
.collect(), pulling partition arrays across the network into the driver terminal:
[('user1', 3)]
Find this content helpful?
☕ Buy me a coffee