Chapter 8: Dataset Engineering for AI

+-------------------------------------------------------------------------------+
|                       DATASET ENGINEERING BEST PRACTICES                      |
+-------------------------------------------------------------------------------+
|  1. Deduplication: MinHash LSH to remove near-duplicate examples              |
|  2. Decontamination: Ensure test/benchmark questions never leak into train    |
|  3. Synthetic Data: Using frontier LLMs to bootstrap edge-case diversity      |
|  4. Complexity Filtering: Remove trivial questions that teach the model zero |
+-------------------------------------------------------------------------------+

Data Quality Trumps Model Architecture

The core tenet of modern AI engineering: the dataset is the model.

If an open-source model makes frequent mistakes on enterprise SQL queries, tweaking the optimizer or transformer architecture rarely helps. The solution is almost always improving the training dataset:

• Removing ambiguous ground-truth labels.
• Ensuring diverse table join patterns.
• Balancing token length distributions.

Synthetic Data Generation & Verification

When human-labeled data is scarce or expensive, AI engineers use frontier models (GPT-4o) to generate training pairs, followed by automated verification filters:

from pydantic import BaseModel
from openai import OpenAI
import instructor

client = instructor.from_openai(OpenAI())

class SyntheticSQLTrainingPair(BaseModel):
    natural_language_question: str
    target_sql_query: str
    difficulty: str # EASY, MEDIUM, HARD
    schema_complexity: str

def generate_synthetic_sql_pair(schema_ddl: str) -> SyntheticSQLTrainingPair:
    prompt = f"""
Generate an authentic, realistic business analytical question and corresponding valid SQL query for this schema.
Include complex aggregations, window functions, or multi-table joins.

SCHEMA:
{schema_ddl}
"""
    return client.chat.completions.create(
        model="gpt-4o",
        response_model=SyntheticSQLTrainingPair,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.8
    )