DataFrame Complex & Nested Type Operations

Parsing JSON structures, querying arrays, and manipulating complex struct elements in PySpark using struct, explode, from_json, and to_json.

What are Complex & Nested Type Operations?

Real-world enterprise data pipelines often ingest complex hierarchical formats like JSON or Avro. Instead of simple flat rows, columns contain nested structures:

• Structs: Nested objects containing their own sub-columns.
• Arrays: Lists containing multiple elements.
• JSON Strings: Raw text fields containing serialized JSON payloads.

PySpark provides native functions (like explode(), from_json(), to_json(), and dot-notation) to parse and query these complex data structures without any custom serialization code.

Core Complex Type Functions

1. Extracting Struct Sub-fields (Dot Notation)

Access nested fields directly using standard dot-notation:

# Access 'street' sub-field nested inside 'address' struct
df.select("address.street", "address.city")

2. Flattening Lists with explode()

Flattens an array column, splitting each array element into its own separate row:

from pyspark.sql import functions as F
# Generates a separate row for each tag inside the 'tags' array column
df.select("post_id", F.explode("tags").alias("tag"))

3. Parsing JSON Strings with from_json()

Parses serialized JSON string columns directly into fully-typed Structs by applying an explicit schema:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType
from pyspark.sql import functions as F

# Define target JSON schema structure
json_schema = StructType([
    StructField("city", StringType()),
    StructField("zip", IntegerType())
])

# Parse column
parsed_df = df.withColumn("location", F.from_json("raw_json_str", json_schema))

Example Usage Pipeline

Below is a complete, copy-paste-ready PySpark script demonstrating nested type operations:

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, ArrayType

# 1. Setup local Spark session
spark = SparkSession.builder \
    .appName("DataFrame Complex Types Demo") \
    .master("local[*]") \
    .getOrCreate()

# 2. Dummy dataset with nested array and raw JSON string columns
data = [
    ("usr_101", ["Python", "Spark"], '{"city": "Mumbai", "country": "India"}'),
    ("usr_102", ["SQL", "Snowflake"], '{"city": "Bangalore", "country": "India"}'),
]
columns = ["user_id", "skills_array", "raw_location_json"]
df = spark.createDataFrame(data, columns)

# 3. Define schema for parsing the raw JSON string
location_schema = StructType([
    StructField("city", StringType(), True),
    StructField("country", StringType(), True)
])

# 4. Perform Complex Transformations:
# - Parse raw JSON column into a structured struct
# - Extract 'city' sub-field from parsed struct
# - Explode 'skills array' so each skill gets its own row
transformed_df = df \
    .withColumn("location_struct", F.from_json(F.col("raw_location_json"), location_schema)) \
    .select(
        "user_id",
        F.col("location_struct.city").alias("city"),
        F.explode("skills_array").alias("skill")
    )

# 5. Show results
print("=== Original Nested DataFrame ===")
df.show(truncate=False)
df.printSchema()

print("=== Flattened & Parsed DataFrame ===")
transformed_df.show(truncate=False)
transformed_df.printSchema()

Rendered Output:

=== Original Nested DataFrame ===
+-------+------------------+-----------------------------------------+
|user_id|skills_array      |raw_location_json                        |
+-------+------------------+-----------------------------------------+
|usr_101|[Python, Spark]    |{"city": "Mumbai", "country": "India"}   |
|usr_102|[SQL, Snowflake]  |{"city": "Bangalore", "country": "India"}|
+-------+------------------+-----------------------------------------+

root
 |-- user_id: string (nullable = true)
 |-- skills_array: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- raw_location_json: string (nullable = true)

=== Flattened & Parsed DataFrame ===
+-------+---------+---------+
|user_id|city     |skill    |
+-------+---------+---------+
|usr_101|Mumbai   |Python   |
|usr_101|Mumbai   |Spark    |
|usr_102|Bangalore|SQL      |
|usr_102|Bangalore|Snowflake|
+-------+---------+---------+

root
 |-- user_id: string (nullable = true)
 |-- city: string (nullable = true)
 |-- skill: string (nullable = true)