LLM Observability, Tracing, and Monitoring

+---------------------------------------------------------------------------------+
|                       DISTRIBUTED TRACE OF A RAG REQUEST                        |
+---------------------------------------------------------------------------------+
|  [User Request] --------------------------------------------------> (Total 1.4s) |
|     ├── [Vector Search: ChromaDB] -------> (120ms)                              |
|     ├── [Prompt Assembly & Guardrail] ---> (15ms)                               |
|     └── [LLM Call: GPT-4o] --------------> (1265ms, 482 Prompt, 85 Comp Tokens) |
+---------------------------------------------------------------------------------+

Why Standard APM Tools (Datadog, New Relic) Are Not Enough

Standard application monitoring only tracks CPU, memory, and HTTP status codes. It is completely blind to:

• Prompt and response payloads.
• Individual token generation latencies (Time to First Token vs. inter-token latency).
• RAG retrieval relevance scores.
• Model hallucination rates and safety guardrail triggers.

Integrating Open-Source Tracing with OpenTelemetry / Langfuse

import os
from langfuse.openai import OpenAI

# Langfuse automatically intercepts and traces OpenAI calls with zero boilerplate
client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),
    # Reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST from env
)

def query_rag_assistant(user_id: str, prompt: str):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        metadata={"user_id": user_id, "environment": "production"},
        tags=["rag", "data_engineering"]
    )
    return response.choices[0].message.content

Golden Metrics for LLMOps

1. TTFT (Time To First Token): Latency before the first token reaches the user. Target: $< 800 ext{ms}$.
2. Token Generation Speed: Output tokens per second. Target: $> 30 ext{ tokens/sec}$.
3. Cache Hit Rate: Percentage of queries answered by semantic cache without model inference. Target: $> 35\%$.
4. Cost Per 1,000 Inferences: Continuous financial tracking of API spend.