RDD - Transformation SortByKey

The sortByKey() transformation is specialized for Key-Value Pair RDDs where each element is a tuple (key, value). It sorts the RDD elements based on the natural ordering of the keys (alphabetical, chronological, or numerical) in either ascending or descending order.

Performance Implications

Sorting requires a Wide Dependency. Spark must shuffle data across the network to group and sort elements by keys. To optimize this, Spark partitions the sorted data using a RangePartitioner, ensuring that elements in partition 1 are smaller than elements in partition 2, which allows sorting to occur independently within each partition block.

PySpark Code Examples

Setup Spark Session

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("RDD Transformation SortByKey") \
    .master("local[*]") \
    .getOrCreate()

sc = spark.sparkContext

Example A: Sorting Alphabetical Keys (Ascending)

Let's sort a list of usernames alphabetically:

# 1. Input RDD of (Username, Score) tuples
user_scores = sc.parallelize([
    ("Charlie", 85),
    ("Alice", 99),
    ("Eve", 92),
    ("Bob", 78)
])

# 2. Sort keys in ascending order (default behavior)
sorted_users_asc = user_scores.sortByKey(ascending=True)

# 3. View final results
print("Sorted Users (Ascending):")
print(sorted_users_asc.collect())
# Output: [('Alice', 99), ('Bob', 78), ('Charlie', 85), ('Eve', 92)]

Example B: Sorting Numerical Keys (Descending)

Let's sort a list of transaction IDs in descending order:

# 1. Input RDD of (TransactionID, PurchaseAmount) tuples
transactions = sc.parallelize([
    (202, 54),
    (101, 120),
    (305, 15),
    (103, 90)
])

# 2. Sort by transaction ID keys descending
sorted_txns_desc = transactions.sortByKey(ascending=False)

print("Sorted Transactions (Descending):")
print(sorted_txns_desc.collect())
# Output: [(305, 15), (202, 54), (103, 90), (101, 120)]