MapReduce Execution Tracing: Web Access Logs
Series
Data Engineering & Distributed Systems Series
Estimated Time
~40 Mins Lab
Lab Objective
Trace raw HTTP web access log records step-by-step through the Map, Combiner, Partitioner, Shuffle/Sort, and Reduce execution stages.
Input Dataset & Parameters
Raw HTTP Logs:
Goal & Cluster Setup:
- Goal: Compute Average Response Time for successful requests (
status_code == 200), grouped byip_address. - Split 1 (Mapper A): Rows 1, 2, 3.
- Split 2 (Mapper B): Rows 4, 5, 6.
- Partitioner:
Partition = ip_address_last_octet % 2(2 Reducers).
Execution Trace Solution
1. Map Phase (Emitting Sum & Count Tuples)
Each mapper reads its assigned log split and emits key-value pairs of (ip_address, (response_time, count)) for successful requests (status_code == 200):
- Mapper A (Split 1): Emits
[("192.168.1.5", (120, 1)), ("192.168.1.5", (95, 1))](Row 2 with 500 status code is filtered out). - Mapper B (Split 2): Emits
[("192.168.1.8", (180, 1))](Rows 4 & 6 with non-200 status codes are filtered out).
2. Combiner Phase (Local Pre-Aggregation)
Aggregates intermediate values locally on each mapper node to minimize network transfer during shuffle:
- Combiner A: Combines IP
192.168.1.5: Sums time120+95=215, Count1+1=2→ Emits("192.168.1.5", (215, 2)). - Combiner B: Emits
("192.168.1.8", (180, 1)).
3. Shuffle & Partitioner Routing Phase
Computes partition routing using last_octet % 2 to distribute intermediate keys to Reducers:
- Reducer 0 (even octet, e.g.,
8 % 2 = 0): Receives("192.168.1.8", [(180, 1)]). - Reducer 1 (odd octet, e.g.,
5 % 2 = 1): Receives("192.168.1.5", [(215, 2)]).
4. Reduce Phase & Final Output
Divides aggregated total response times by response counts to output final averages:
- Reducer 0 Final Output:
("192.168.1.8", 180.0 ms)(Calculated as 180ms / 1 request). - Reducer 1 Final Output:
("192.168.1.5", 107.5 ms)(Calculated as 215ms / 2 requests).
Final Results Table
Find this content helpful?
☕ Buy me a coffee