RDD - Action Reduce

The reduce() action aggregates all elements in an RDD using a specified commutative and associative binary operator (a function taking two arguments and returning one). It iteratively collapses the dataset until only a single scalar value remains, which is returned to the Driver.

Important Rules for reduce()

1. Associative: The order in which operations are grouped must not affect the result:

(A + B) + C = A + (B + C)

1. Commutative: The order of the operands must not affect the result:

A + B = B + A

• Why?: Spark runs the reduction in parallel locally on separate partitions first, and then combines those partition-level results. Non-associative or non-commutative operations (like division or subtraction) will yield inconsistent, random results when executed in parallel!

PySpark Code Examples

Setup Spark Session

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("RDD Action Reduce") \
    .master("local[*]") \
    .getOrCreate()

sc = spark.sparkContext

Example A: Summing Elements

Let's sum all the numbers in an RDD:

# 1. Create RDD
numbers = sc.parallelize([1, 2, 3, 4, 5])

# 2. Reduce using addition
total_sum = numbers.reduce(lambda x, y: x + y)

print("Total Sum:", total_sum)
# Output: Total Sum: 15

Example B: Finding Global Extremes (Max and Min)

Let's find the maximum temperature from a distributed sensor dataset:

# 1. Temperature logs RDD
temperatures = sc.parallelize([22.5, 31.2, 19.8, 35.4, 28.1])

# 2. Reduce to find the maximum value
max_temp = temperatures.reduce(lambda t1, t2: t1 if t1 > t2 else t2)

print("Maximum Temperature:", max_temp)
# Output: Maximum Temperature: 35.4

Example C: String Concatenation

Let's merge a list of strings together into a single phrase:

words = sc.parallelize(["Apache", "Spark", "is", "Fast"])

# Concatenate words with a space
sentence = words.reduce(lambda word1, word2: f"{word1} {word2}")

print("Concatenated Sentence:", sentence)
# Output: Concatenated Sentence: Apache Spark is Fast