RDD - Action SaveAsTextFile

The saveAsTextFile() action writes the elements of an RDD to a filesystem, HDFS directory, or cloud bucket (S3, Azure Blob) as plain text files. It is the primary production standard for exporting transformed and clean ETL data outputs safely and in a fully distributed manner.

Internal Writing Mechanics

When saveAsTextFile(path) is triggered:

1. Spark does not collect data to the Driver program.
2. Instead, each executor node writes its partitions directly to the physical disk storage at the specified path.
3. Spark creates a directory at the destination path containing:
   • An empty success flag file named _SUCCESS.
   • Individual partition file outputs named part-00000, part-00001, part-00002 (one file per RDD partition).

graph TD
    subgraph Storage["Storage Target (e.g. output_logs/)"]
        Success["_SUCCESS"]
        P0["part-00000 (Partition 1 Data)"]
        P1["part-00001 (Partition 2 Data)"]
    end

    subgraph Cluster["Executors (Worker Nodes)"]
        E1["Executor 1 (Partition 1)"] -->|Write direct| P0
        E2["Executor 2 (Partition 2)"] -->|Write direct| P1
    end

    style Storage fill:#fff3e0,stroke:#e65100,stroke-width:2px;
    style Cluster fill:#efebe9,stroke:#8d6e63,stroke-width:2px;

PySpark Code Example

Setup Spark Session

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("RDD Action SaveAsTextFile") \
    .master("local[*]") \
    .getOrCreate()

sc = spark.sparkContext

In Action: Exporting RDD Data

Let's export clean transaction records directly to a local directory:

# 1. Create RDD with 3 elements split into 2 partitions
cleaned_users_rdd = sc.parallelize([
    "ID101,Alice,Active",
    "ID102,Bob,Inactive",
    "ID103,Charlie,Active"
], numSlices=2)

# 2. Save RDD to a directory named 'output cleaned users'
# Note: This will create a directory containing part-00000 and part-00001
cleaned_users_rdd.saveAsTextFile("output_cleaned_users")

print("Data exported successfully.")