LLM-Based Data Cataloging & Metadata Generation

+-----------------------------------------------------------------------+
|                    AUTOMATED METADATA ENRICHMENT                      |
+-----------------------------------------------------------------------+
|  Table Schema + Sample 100 Rows  -> LLM Cataloger -> Rich Markdown Docs|
|                                                          |            |
|                                                          v            |
|  dbt docs / DataHub / Atlan      <- Verified Docs <- Review Dashboard |
+-----------------------------------------------------------------------+

The Data Catalog Problem

Over 80% of enterprise data warehouse tables lack documentation, descriptions, or ownership tags. Data engineers simply don't have the time to manually write docstrings for thousands of staging and mart tables.

LLMs can inspect:

1. Column names and data types.
2. Distinct value profiles and statistical samples.
3. Upstream and downstream lineage. And automatically synthesize crystal-clear, business-ready column descriptions.

Generating dbt schema.yml Automatically

import pandas as pd
from openai import OpenAI

client = OpenAI()

def generate_dbt_documentation(table_name: str, sample_df: pd.DataFrame) -> str:
    summary_info = {
        "table": table_name,
        "columns": {}
    }
    for col in sample_df.columns:
        summary_info["columns"][col] = {
            "dtype": str(sample_df[col].dtype),
            "sample_values": sample_df[col].dropna().head(3).tolist(),
            "null_count": int(sample_df[col].isnull().sum()),
            "unique_count": int(sample_df[col].nunique())
        }

    prompt = f"""
Generate a production dbt schema.yml file documenting this table and its columns.
Write concise, professional descriptions for each column and recommend appropriate dbt tests (e.g. not_null, unique, accepted_values).

TABLE PROFILE:
{summary_info}
"""
    res = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2
    )
    return res.choices[0].message.content