CI/CD Pipelines for LLM Applications

+---------------------------------------------------------------------------------+
|                       CONTINUOUS INTEGRATION FOR AI (CI)                        |
+---------------------------------------------------------------------------------+
|  1. Developer modifies prompt.py or rag_retriever.py in PR                      |
|  2. GitHub Action triggers:                                                     |
|     - Run Unit Tests (Syntax, Pydantic Schema Validation)                       |
|     - Run Automated Evals: 50 golden benchmark queries                          |
|     - LLM-as-a-Judge scores accuracy and hallucination rate                     |
|  3. PR Gated: Block merge if overall accuracy drops below 95%                   |
+---------------------------------------------------------------------------------+

Continuous Evaluation in GitHub Actions

Just as data engineers run pytest and dbt test before deploying data models, LLM systems require continuous evaluation to prevent prompt regressions.

Production GitHub Actions Workflow (eval.yml)

name: LLM Prompt & Pipeline Evaluation

on:
  pull_request:
    paths:

      - 'src/prompts/**'
      - 'src/rag/**'

jobs:
  run-evals:
    runs-on: ubuntu-latest
    steps:

      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Dependencies
        run: pip install -r requirements-eval.txt

      - name: Execute Evaluation Suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python tests/eval_prompts.py --golden-set benchmarks/golden_sql_tests.json --threshold 0.95