Chapter 3: Evaluation Methodology

+-------------------------------------------------------------------------------+
|                    THE 3 LEVELS OF AI APPLICATION EVALUATION                  |
+-------------------------------------------------------------------------------+
|  1. Component Evaluation | Test vector retrieval recall, prompt latency, AST  |
|  2. End-to-End Evals     | Ground-truth benchmark test suites (Golden sets)   |
|  3. Production Monitoring| User thumbs-up/down, session drop-off, cost/token  |
+-------------------------------------------------------------------------------+

Why LLM Evaluation is Notoriously Difficult

In traditional software, tests are binary assertions: assert calculate_tax(100) == 10.

In foundation model applications:

1. Open-Endedness: There are thousands of valid ways to summarize a data pipeline incident.
2. Non-Determinism: Minor prompt changes or temperature variations shift wording.
3. Subjectivity: Tone, conciseness, and helpfulness vary by user preference.

Building a "Golden Evaluation Dataset"

Chip emphasizes that the highest ROI activity in AI engineering is creating a Golden Dataset (50 to 500 realistic, representative query-response pairs):

{
  "test_id": "eval_042",
  "user_query": "How do I backfill an Airflow DAG for March 2026?",
  "reference_context": "airflow dags backfill -s 2026-03-01 -e 2026-03-31 dag_id",
  "expected_key_facts": [
    "CLI command airflow dags backfill",
    "Start date parameter -s",
    "End date parameter -e"
  ],
  "negative_constraints": [
    "Do not recommend modifying execution_date manually in database"
  ]
}

Core Metrics for Retrieval & Generation

• Retrieval Precision & Recall: Did the vector search find the exact documents containing the answer?
• Faithfulness (Groundedness): Can every claim in the generated answer be traced to the retrieved context? (Catches hallucinations).
• Answer Relevance: Did the model directly address the user's intent without wandering off-topic?