Databricks Q&A (Questions & Answers)

This section provides technical questions and detailed answers designed to deepen your understanding of Databricks architecture, administration, and development best practices.

1. Core Architecture & Delta Lake

Q1: What is the difference between Z-Ordering and Liquid Clustering in Delta Lake? When should you use each?

• Z-Ordering:
• How it works: It is a multidimensional clustering technique that co-locates related information in the same set of files. It reorganizes data along specified columns.
• Limitations: Z-Ordering is expensive to compute, is not incremental (you have to re-cluster the entire dataset/partition), and performance degrades if you cluster on more than 2–3 columns. It also requires you to partition data first, which can lead to overpartitioning.
• Liquid Clustering:
• How it works: Introduced in Delta Lake 3.0, it simplifies data layout by clustering data dynamically without relying on fixed hive-style partition columns. It uses a flexible, multi-dimensional clustering key and organizes data incrementally as it is written.
• Advantages: It is fast, supports incremental clustering (avoiding full-table rewrites), and allows you to redefine clustering keys without rewriting historical data.
• When to use what:
• Use Liquid Clustering for all new tables in Databricks (Runtime 13.3 LTS and above). It is the recommended standard.
• Use Z-Ordering only when working on older Databricks runtimes or legacy pipelines that do not support Liquid Clustering.

Q2: Explain the purpose of Delta Lake's OPTIMIZE and VACUUM commands. What is the safety threshold for VACUUM?

• OPTIMIZE:
• Coalesces small files into larger, optimal-sized files (typically 1GB) to improve query read performance (solving the "small file problem").
• Can be combined with ZORDER BY (for legacy tables) to cluster data layout.
• VACUUM:
• Recursively vacuums directories associated with the Delta table to remove data files that are no longer in the active state of the table and are older than a retention threshold.
• This is critical for reclaiming storage space and complying with GDPR/CCPA requests (deleting physical data, not just logically marking it deleted).
• Safety Threshold:
• By default, VACUUM retains files up to 7 days (168 hours).
• Crucial Rule: Running VACUUM with a retention threshold of 0 hours or any duration shorter than the default is dangerous because concurrent writers or active readers might be accessing those files, leading to job failures. You must set spark.databricks.delta.vacuum.parallelDelete.enabled or spark.databricks.delta.retentionDurationCheck.enabled = false to override the check, which should only be done with extreme caution.

Q3: How do Schema Enforcement and Schema Evolution differ in Delta Lake?

• Schema Enforcement (Schema Validation):
• Preventative measure. Delta Lake validates that any write to a table matches the table's pre-defined schema.
• If a write contains columns not present in the table, Delta Lake rejects the transaction with an error (preventing bad data from polluting the table).
• Schema Evolution:
• Adaptability measure. Allows users to change a table's current schema to accommodate data changes over time.
• Activated by adding the .option("mergeSchema", "true") option to your Spark write command, or by running an ALTER TABLE statement. This adds new columns automatically while keeping existing columns intact.

2. Infrastructure & Compute

Q4: Compare Databricks Serverless Compute with Classic (Customer-Managed) Compute.

Q5: How do Databricks Asset Bundles (DABs) improve on standard Terraform deployments for Databricks resources?

• Terraform is a general-purpose Infrastructure as Code (IaC) tool. While it can deploy Databricks assets using the Databricks Terraform Provider, it requires deep knowledge of Terraform syntax, state file management, and backend configurations.
• Databricks Asset Bundles (DABs) are developer-centric and tailored specifically for Databricks. DABs:
• Provide a simple YAML-based syntax (databricks.yml) to define notebooks, libraries, workflows, and pipelines together as a single project bundle.
• Auto-manage environment configurations (e.g., dev, staging, prod) natively.
• Integrate directly with the Databricks CLI and local IDEs (like VS Code), enabling one-command validation and deployments (databricks bundle deploy).
• Under the hood, DABs can generate and manage Terraform configurations, combining the ease of developer tools with the stability of Terraform.

3. Governance & Unity Catalog

Q6: How does Unity Catalog enable dynamic row-level filtering and column-level masking?

Unity Catalog lets you apply fine-grained access control using standard SQL UDFs (User Defined Functions):

• Column Masking:
• You define a SQL function that determines what a user sees based on their identity or group membership (e.g., using IS_MEMBER('admins')).
• If a non-admin queries the table, the function replaces the sensitive data with REDACTED or asterisks (****).
• The mask is attached to the column definition: ALTER TABLE users ALTER COLUMN ssn SET MASK redact_ssn_fn;.
• Row-Level Filtering:
• You define a filter policy function that returns a boolean.
• When a user queries the table, Unity Catalog appends the filter condition under the hood.
• For example, a filter might restrict rows based on region: RETURN region = CURRENT_USER_REGION().
• The filter is bound to the table: ALTER TABLE transactions SET ROW FILTER region_filter_fn ON (region);.

Q7: What are the key steps to upgrade a legacy workspace (Hive Metastore) to Unity Catalog?

1. Configure the Metastore: Create a Unity Catalog metastore in the account console and attach it to your workspaces.
2. Setup Access Connectors: Provision a cloud service identity (e.g., AWS IAM Role or Azure Managed Identity) with permissions to read/write storage containers.
3. Define External Locations: In Unity Catalog, create Storage Credentials and External Locations pointing to the S3 buckets / ADLS Gen2 containers.
4. Upgrade Tables:
5. For external tables: Use SYNC command or Databricks UI upgrade wizard to register legacy tables into Unity Catalog without moving physical data.
6. For managed tables: Upgrade by copying data using CREATE TABLE ... AS SELECT (CTAS) into the new Unity Catalog workspace catalog.
7. Assign Privileges: Grant permissions to users and groups using Unity Catalog's standard SQL GRANT statements.

4. Orchestration & Workflows

Q8: What is the difference between Delta Live Tables (DLT) and standard Databricks Workflows?

• Databricks Workflows:
• An orchestrator of tasks. It runs notebooks, JARs, Python scripts, dbt projects, or DLT pipelines in a specified order (using a DAG).
• You define how and when to execute the steps, but you are responsible for managing state, checkpoints, cluster startup, and coding error-handling logic within the notebook.
• Delta Live Tables (DLT):
• A declarative framework for building reliable, maintainable, and testable data processing pipelines.
• You write SQL or Python queries to define target tables, and DLT manages cluster orchestration, task dependencies, data quality checks (expectations), recovery, and schema evolution automatically.
• Summary: DLT is a tool for building the data pipelines themselves; Databricks Workflows is the parent scheduler that triggers and manages DLT pipelines alongside other jobs.