Book Case Study: M&M Count Aggregation
Series
Data Engineering & Distributed Systems Series
Estimated Time
~25 Mins Read
Case Study Objective
Analyze the classic M&M count example from Learning Spark (2nd Edition) — demonstrating DataFrame ingestion, grouping, aggregation, sorting, and predicate pushdown.
PySpark Code Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, col, desc
# 1. Initialize SparkSession
spark = SparkSession.builder.appName("MnMCount").getOrCreate()
# 2. Ingest CSV dataset
mnm_df = spark.read.format("csv") \
.option("header", "true") \
.option("inferSchema", "true") \
.load("mnm_dataset.csv")
# 3. Filter for California & aggregate counts grouped by State and Color
ca_count_df = mnm_df \
.select("State", "Color", "Count") \
.filter(col("State") == "CA") \
.groupBy("State", "Color") \
.agg(sum("Count").alias("Total")) \
.orderBy(desc("Total"))
# Show top 10 results
ca_count_df.show(10, False)
Execution Pipeline Mechanics
graph TD
A["mnm_dataset.csv (Ingestion)"] --> B["Select: State, Color, Count"]
B --> C["Filter: State == 'CA' (Predicate Pushdown)"]
C --> D["Group By: State, Color (Shuffle Phase)"]
D --> E["Aggregate: sum(Count) as Total"]
E --> F["Order By: Total DESC"]
F --> G["Action: show() (Triggers Execution Job)"]
style C fill:#eff6ff,stroke:#2563eb,stroke-width:2px;
style D fill:#fef2f2,stroke:#dc2626,stroke-width:2px;
style G fill:#f0fdf4,stroke:#16a34a,stroke-width:2px;
Find this content helpful?
☕ Buy me a coffee