CSV Ingestion
Comma-Separated Values (CSV) is one of the most common file formats in data engineering. Because CSV is a plain-text, row-oriented format, it does not store schema definitions, column names, or data types natively. Spark provides a rich set of options to read and write CSV files safely and efficiently.
Key CSV Reader Options
When calling spark.read.format("csv"), you can chain multiple .option("key", "value") calls to configure the reader:
PySpark Code Example: Reading & Writing CSV
Here is a complete, copy-paste-ready script showing how to ingest raw CSV files (including multi-line data) and save DataFrames back as CSVs with compression:
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
# 1. Setup Spark Session
spark = SparkSession.builder \
.appName("CSV Ingestion") \
.master("local[*]") \
.getOrCreate()
# 2. Define schema explicitly (Production Best Practice)
csv_schema = StructType([
StructField("transaction_id", IntegerType(), False),
StructField("product_name", StringType(), True),
StructField("price", IntegerType(), True),
StructField("customer_comments", StringType(), True)
])
# 3. Read a CSV file with options
# We enable multiLine because customer comments can have embedded line breaks
df = spark.read \
.format("csv") \
.option("header", "true") \
.option("sep", ",") \
.option("nullValue", "NA") \
.option("multiLine", "true") \
.schema(csv_schema) \
.load("dataset.csv")
df.show()
# 4. Write data back as a compressed CSV
# We will use gzip compression to save storage space
df.write \
.format("csv") \
.mode("overwrite") \
.option("header", "true") \
.option("compression", "gzip") \
.save("output_csv_directory")
Find this content helpful?
☕ Buy me a coffee