Delta Time Travel

The magical ability of Delta Lake to query older versions of a table is called Time Travel. This is made possible by the Delta Log (transaction log)a detailed record of every single transaction committed to the table.

How the Delta Log Works

Every time you write, update, or delete records in a Delta table, Delta:

1. Writes new parquet files containing the updated partitions on disk.
2. Appends a new commit file inside the _delta_log/ directory (e.g. 000000.json, 000001.json).
3. These JSON transaction logs record exactly which files were added and which files were logically deleted during that transaction.

When you query version 1, Spark reads only the active parquet files associated with commit 000001.json, ignoring later files!

Time Travel Query Methods

Spark SQL supports querying historical states using two different approaches:

1. Version-Based Query (versionAsOf)

Specify the integer version number of the transaction:

df = spark.read \
    .format("delta") \
    .option("versionAsOf", 2) \
    .load("delta_storage_path")

2. Timestamp-Based Query (timestampAsOf)

Specify a date or timestamp string to query the data exactly as it existed at that time:

df = spark.read \
    .format("delta") \
    .option("timestampAsOf", "2026-05-23 10:00:00") \
    .load("delta_storage_path")

PySpark Code Example: Historical Queries & Rollbacks

Here is a complete script demonstrating how to inspect table history, query older versions, and perform table rollbacks:

from pyspark.sql import SparkSession
from delta.tables import DeltaTable

# 1. Setup Spark configured with Delta Lake
spark = SparkSession.builder \
    .appName("Delta Time Travel") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .master("local[*]") \
    .getOrCreate()

# 2. Inspect table transaction history
deltaTable = DeltaTable.forPath(spark, "delta_storage_directory")
history_df = deltaTable.history()

# Show the transaction logs, timestamps, user details, and operational metrics
history_df.select("version", "timestamp", "operation", "operationParameters").show(truncate=False)

# 3. Query the Original Version 0 (Before any Updates/Upserts occurred)
original_df = spark.read \
    .format("delta") \
    .option("versionAsOf", 0) \
    .load("delta_storage_directory")

print("Original Table (Version 0):")
original_df.show()

# 4. Perform a Table Rollback (Restoring version 0 as the active state)
# To rollback, we simply read the target version and overwrite the active table path!
original_df.write \
    .format("delta") \
    .mode("overwrite") \
    .save("delta_storage_directory")

print("Rolled Back Table (Active Table is now matching Version 0):")
spark.read.format("delta").load("delta_storage_directory").show()