Chapter 6: Retrieval-Augmented Generation & Agents

+---------------------------------------------------------------------------------+
|                               ADVANCED RAG PIPELINE                             |
+---------------------------------------------------------------------------------+
|  User Query -> Query Expansion / HyDE -> Hybrid Search (BM25 + Dense Vector)   |
|                                                       |                         |
|                                                       v                         |
|  Answer <- Generator Model <- Context Fusion <- Cross-Encoder Reranker          |
+---------------------------------------------------------------------------------+

Beyond Naive RAG

Naive RAG (Chunk $ o$ Embed $ o$ Top-K $ o$ LLM) frequently fails in production. Chip outlines the advanced techniques required for enterprise-grade retrieval:

1. Hybrid Search (Dense + Sparse)

Dense embeddings (e.g., OpenAI, Cohere) excel at conceptual meaning, but fail on exact part numbers, error codes, and UUIDs (ERR_99812_B).

• Dense Retrieval: Semantic similarity via vector cosine distance.
• Sparse Retrieval: BM25 keyword matching for exact IDs and acronyms.
• Reciprocal Rank Fusion (RRF): Merging both ranking lists into a single superior ranking.

2. Cross-Encoder Reranking

Vector search is fast ($O(1)$ with HNSW indexes), but computes similarity on isolated vectors. A cross-encoder reranker (e.g. Cohere Rerank, BGE-Reranker) passes the query and document together through full cross-attention layers, boosting retrieval accuracy by 20-35%.

From RAG to Autonomous Agents

An agent extends RAG with a dynamic feedback loop:

• Observation $ o$ Thought $ o$ Action (Tool Call) $ o$ Observation: The model decides when to search, what tools to call, and inspects its own results to determine if more data is needed.