RDD - Read files (text , csv ..)

While parallelizing local memory collections is useful for testing, real-world big data workflows require reading datasets from external storage systemssuch as a local filesystem, Hadoop Distributed File System (HDFS), Amazon S3, or Azure Blob Storage.

This guide details the core methods for reading files into RDDs (sc.textFile and sc.wholeTextFiles) and provides complete code examples for parsing unstructured raw text, semi-structured CSV tables, and structured JSON logs.

1. Core Spark File-Reading Methods

Spark provides two main low-level methods in SparkContext for loading files into RDDs:

A. sc.textFile(path, minPartitions=None)

• Behavior: Reads a text file line-by-line. Each line of the text file becomes a single element (string) in the resulting RDD.
• Partitions: Spark splits the file across partitions. By default, it allocates one partition per 128MB block of the file, but you can override this by passing a custom minPartitions value.
• Compression: Automatically detects and decompresses standard formats like .gz, .bz2, and .zip on the fly.

B. sc.wholeTextFiles(path, minPartitions=None)

• Behavior: Reads a directory containing multiple small text files. It returns a Key-Value RDD where:
  • Key: The absolute path/URI of the file (string).
  • Value: The complete, entire text content of that specific file (string).
• When to use: Highly optimized for directories containing hundreds of small log or configuration files where you need to process each file as a single record.

2. PySpark Code Examples

Setup Spark Session

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Day01 RDD Read Files") \
    .master("local[*]") \
    .getOrCreate()

sc = spark.sparkContext

A. Reading a Plain Text File (textFile)

Let's read a text file where each element in the RDD corresponds to one line of the file:

# 1. Read a local text file into an RDD
# (Assume a file named 'sample logs.txt' exists in the current directory)
logs_rdd = sc.textFile("sample_logs.txt")

# 2. Print RDD type
print("RDD Type:", type(logs_rdd))
# Output: RDD Type: <class 'pyspark.rdd.RDD'>

# 3. Fetch the first 3 lines of the file using take()
lines_preview = logs_rdd.take(3)
print("First 3 lines of file:")
for line in lines_preview:
    print(f"  Line: {line}")

B. Reading Multiple Files with Wildcards (*)

You can use standard directory globbing wildcards to read all text files matching a pattern into a single RDD:

# Read all files ending in '.log' from the logs directory
all_logs_rdd = sc.textFile("logs/*.log")

# Count total combined lines across all matching log files
total_lines = all_logs_rdd.count()
print(f"Total lines in all log files: {total_lines}")

C. Processing Directories of Small Files (wholeTextFiles)

Let's read an entire directory of small files and inspect the absolute paths along with the file contents:

# 1. Read directory as key-value pairs (FilePath, CompleteContent)
small_files_rdd = sc.wholeTextFiles("configs_directory/")

# 2. View the count of files read
print(f"Number of files loaded: {small_files_rdd.count()}")

# 3. Inspect the records
files_data = small_files_rdd.collect()
for file_path, file_content in files_data[:2]: # Show first 2 files
    print(f"
File Location: {file_path}")
    print(f"File Contents:
{file_content}")

D. Reading and Manually Parsing a CSV File

Because RDD is a low-level API, it does not automatically detect columns or headers of a CSV. We must read the file line-by-line and manually parse the fields:

Let's assume the CSV file dataset.csv contains:

id,name,age,city
1,Alice,28,New York
2,Bob,32,San Francisco
3,Charlie,22,Chicago

# 1. Load the raw text file
raw_csv_rdd = sc.textFile("dataset.csv")

# 2. Filter out the header row
header = raw_csv_rdd.first() # Get the very first line (the header)
data_rows_rdd = raw_csv_rdd.filter(lambda line: line != header)

# 3. Manually split each line by comma and map to a structured tuple: (ID, Name, Age, City)
def parse_csv_line(line):
    parts = line.split(",")
    # Convert types where necessary
    row_id = int(parts[0])
    name = parts[1]
    age = int(parts[2])
    city = parts[3]
    return (row_id, name, age, city)

parsed_rdd = data_rows_rdd.map(parse_csv_line)

# 4. View parsed results
print("Parsed CSV Rows:")
for row in parsed_rdd.collect():
    print(f"  Row: {row}")

# Expected Output:
# Parsed CSV Rows:
# Row: (1, 'Alice', 28, 'New York')
# Row: (2, 'Bob', 32, 'San Francisco')
# Row: (3, 'Charlie', 22, 'Chicago')

E. Reading and Parsing JSON Lines

To read standard JSON Lines files (where each line is a valid JSON object), we can use Python's built-in json module:

Let's assume data.json contains:

{"user": "Alice", "score": 95}
{"user": "Bob", "score": 88}

import json

# 1. Read JSON file line-by-line
raw_json_rdd = sc.textFile("data.json")

# 2. Use map to parse the JSON string into a Python dictionary
parsed_json_rdd = raw_json_rdd.map(lambda line: json.loads(line))

# 3. Extract specific values from the parsed dictionaries
user_scores_rdd = parsed_json_rdd.map(lambda obj: (obj["user"], obj["score"]))

print("Parsed JSON records:")
print(user_scores_rdd.collect())
# Output: [('Alice', 95), ('Bob', 88)]