Parquet Columnar Format

Apache Parquet is an open-source, binary, columnar storage format designed for high-performance large-scale analytics. It is the default file format for Apache Spark SQL and modern data lakehouse architectures.

Row-Oriented vs. Columnar Storage

Traditional file formats (like CSV and JSON) store data in a row-by-row structure. Columnar formats group data on disk by columns instead.

graph TD
    subgraph RowOriented["Row-Oriented (CSV/JSON) - Stores record blocks consecutively"]
        direction LR
        R1["Row 1: Name, Salary, Dept"] --> R2["Row 2: Name, Salary, Dept"] --> R3["Row 3: Name, Salary, Dept"]
    end
    subgraph Columnar["Columnar (Parquet) - Stores column values consecutively"]
        direction LR
        C1["Col 1: Name1, Name2, Name3"] --> C2["Col 2: Salary1, Salary2, Salary3"] --> C3["Col 3: Dept1, Dept2, Dept3"]
    end
    style RowOriented fill:#ffebee,stroke:#c62828,stroke-width:2px;
    style Columnar fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;

Why Columnar Storage is Better for Analytics

1. Projection Pruning (Column Selection): If your query is SELECT salary FROM employees, Spark only reads the bytes containing the salary column from disk, skipping the names and departments entirely. This reduces disk I/O by up to 90%!
2. High Compression Ratio: Because identical data types sit next to each other on disk (e.g. integer salary list), compression algorithms (like Snappy or Gzip) operate at maximum efficiency, saving up to 75% of storage space.
3. Predicate Pushdown (Filter at Storage Level): Parquet files are divided into "Row Groups", each storing min/max metadata statistics for each column. If your query is WHERE salary > 100000, Spark reads the metadata of the row group first. If the max salary in that group is 90000, Spark skips reading the entire row group, avoiding massive disk reads!
4. Self-Describing: Parquet files store table schemas and column names internally inside the file footer metadata. Spark doesn't need schema inference.

PySpark Code Example: Reading & Partitioned Writing

Here is a complete script demonstrating how to read and write Parquet files with custom partition directories:

from pyspark.sql import SparkSession

# 1. Setup Spark
spark = SparkSession.builder \
    .appName("Parquet Formats") \
    .master("local[*]") \
    .getOrCreate()

# 2. Sample Data
data = [
    ("Alice", 90000, "Engineering"),
    ("Bob", 60000, "Marketing"),
    ("Charlie", 95000, "Engineering"),
    ("David", 50000, "Sales")
]
columns = ["name", "salary", "department"]
df = spark.createDataFrame(data, columns)

# 3. Write DataFrame as a Parquet dataset
# Snappy is Spark's default high-speed compression codec
df.write \
    .format("parquet") \
    .mode("overwrite") \
    .option("compression", "snappy") \
    .save("output_parquet_directory")

# 4. Read Parquet files (Schema is automatically resolved!)
parquet_df = spark.read \
    .format("parquet") \
    .load("output_parquet_directory")

parquet_df.printSchema()
parquet_df.show()

# 5. Write Parquet with Partitioning
# This creates subdirectory folders like: output partitioned/department=Engineering/
df.write \
    .format("parquet") \
    .mode("overwrite") \
    .partitionBy("department") \
    .save("output_partitioned")