RDD - Transformation Map

The map() transformation is one of the most fundamental operations in Apache Spark. It takes an existing RDD, applies a user-defined function to each individual element, and returns a new RDD containing the results.

Key Characteristics

• 1-to-1 Mapping: The number of input elements in the parent RDD is exactly equal to the number of output elements in the child RDD.
• Narrow Dependency: It does not require moving data across the network (shuffling). Each partition is processed locally on its executor node, making it extremely fast.
• Type Safety / Flexibility: The output elements can have a different data type than the input elements (e.g., mapping strings to integers).

PySpark Code Examples

Setup Spark Session

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("RDD Transformation Map") \
    .master("local[*]") \
    .getOrCreate()

sc = spark.sparkContext

Example A: Modifying Numeric Values

Let's double every integer inside a distributed RDD:

# 1. Create a raw RDD
numbers = sc.parallelize([1, 2, 3, 4, 5])

# 2. Apply map transformation (lazy)
doubled_numbers = numbers.map(lambda x: x * 2)

# 3. Trigger action to see results
print("Doubled Numbers:", doubled_numbers.collect())
# Output: Doubled Numbers: [2, 4, 6, 8, 10]

Example B: Extracting Fields from Key-Value Strings

Let's parse raw click logs and extract only the username:

# 1. Raw log lines RDD
raw_logs = sc.parallelize([
    "user_alice,login,2026-05-23",
    "user_bob,click,2026-05-23",
    "user_charlie,logout,2026-05-23"
])

# 2. Map to extract only the username (first field before comma)
usernames = raw_logs.map(lambda log: log.split(",")[0])

# 3. Fetch results
print("Usernames:", usernames.collect())
# Output: Usernames: ['user alice', 'user bob', 'user charlie']

Example C: Type Conversion (String to Length)

Let's map a list of strings to their respective lengths:

# 1. RDD of strings
words = sc.parallelize(["Spark", "RDD", "Map", "Transformation"])

# 2. Map strings to a tuple containing the word and its length
word_lengths = words.map(lambda word: (word, len(word)))

print("Word Lengths:", word_lengths.collect())
# Output: Word Lengths: [('Spark', 5), ('RDD', 3), ('Map', 3), ('Transformation', 14)]