Evaluating Fine-Tuned Models

+-----------------------------------------------------------------------------+
|                      EVALUATION PYRAMID FOR FINE-TUNING                     |
+-----------------------------------------------------------------------------+
|  Level 3: Human Domain Expert Review (Ground Truth)                        |
|  Level 2: LLM-as-a-Judge (GPT-4o evaluating alignment & compliance)        |
|  Level 1: Deterministic Benchmarks (Exact Match, Code Execution, SQL Lint) |
|  Level 0: Training Loss & Perplexity (Basic sanity check)                  |
+-----------------------------------------------------------------------------+

Why Loss Curves Lie

A declining training loss curve only proves the model is memorizing token transitions in your dataset. It does not prove the model has generalized:

• If training loss drops to near zero while validation loss spikes, the model is overfitting.
• A model with low cross-entropy loss may still output hallucinated table names or invalid syntax.

Automated Evaluation with LLM-as-a-Judge

Using a frontier model (like GPT-4o or Claude 3.5 Sonnet) with a structured grading rubric is the industry standard for scalable evaluation:

from pydantic import BaseModel, Field
from openai import OpenAI

class EvaluationVerdict(BaseModel):
    syntax_correctness_score: int = Field(ge=1, le=5)
    business_logic_score: int = Field(ge=1, le=5)
    explanation: str
    is_safe_to_deploy: bool

def grade_generated_pyspark(prompt: str, ground_truth: str, candidate_output: str) -> EvaluationVerdict:
    client = OpenAI()
    rubric = f"""
You are an expert PySpark evaluator. Grade the candidate model output against the prompt and ground truth.
Scores are 1 (Worst) to 5 (Flawless).

PROMPT:
{prompt}

GROUND TRUTH:
{ground_truth}

CANDIDATE OUTPUT:
{candidate_output}
"""
    res = client.beta.chat.completions.parse(
        model="gpt-4o",
        response_format=EvaluationVerdict,
        messages=[{"role": "user", "content": rubric}]
    )
    return res.choices[0].message.parsed