Chunking Strategies: The Foundation of RAG Retrieval

In Retrieval-Augmented Generation, your retrieval quality is strictly bounded by your chunking strategy.

If chunks are too large (e.g., 4,000 tokens):

• Key factual details are drowned in noise.
• Context windows fill up quickly, driving up API costs.
• The model encounters the "Lost in the Middle" attention degradation.

If chunks are too small (e.g., 50 tokens):

• Semantic context is severed (a SQL WHERE clause is separated from its SELECT table).
• The vector embedding represents an incomplete thought, hurting cosine retrieval accuracy.

1. Chunking Strategies Compared

flowchart TD
    subgraph ChunkingTaxonomy["Chunking Strategy Taxonomy"]
        Fixed["1. Fixed-Size Chunking<br/>Split every N characters/tokens with overlap"]
        Recursive["2. Recursive Character Chunking<br/>Split hierarchically by paragraphs, lines, spaces"]
        Document["3. Document-Aware Chunking<br/>Split by Markdown headers, HTML sections, Code ASTs"]
        Semantic["4. Semantic Distance Chunking<br/>Split dynamically where embedding similarity drops"]

        Fixed --> Recursive --> Document --> Semantic
    end

    style Fixed fill:#475569,color:#fff
    style Recursive fill:#2563eb,color:#fff
    style Document fill:#7c3aed,color:#fff
    style Semantic fill:#059669,color:#fff

1. Fixed-Size Chunking

Splits text every $N$ tokens with a fixed overlap of $M$ tokens (e.g., 500 tokens with 50 tokens overlap).

• Pros: Fast, simple, constant memory size.
• Cons: Cuts words, sentences, or SQL clauses in half.

2. Recursive Character Text Splitting (The Industry Standard)

Attempts to split on paragraph breaks (`

) first. If a paragraph is still larger than the chunk size, it splits on single line breaks ( ). If still too large, on spaces ( `), and finally on raw characters.

• Pros: Preserves complete paragraphs and coherent thoughts.
• Best for: Technical articles, documentation, books.

3. Document-Aware / Markdown Chunking

Uses document structure as natural boundaries:

• In Markdown: Splits by # Header 1, ## Header 2, ### Header 3.
• In Code: Splits by Python functions (def ...) or classes (class ...) using Abstract Syntax Trees (AST).
• In SQL: Splits by CTEs (WITH ... AS) or individual statement boundaries.
• Pros: Preserves complete logical units (e.g., an entire function or section).

4. Semantic Chunking

Calculates embedding vectors for consecutive sentences. When the cosine distance between sentence $i$ and sentence $i+1$ exceeds a threshold (indicating a shift in topic), a chunk boundary is placed.

• Pros: Perfectly adapts chunk size to thematic shifts in text.
• Cons: Requires running an embedding model on every sentence during ingestion (higher compute cost).

2. Handling Tabular Data and SQL Schemas

Data engineers frequently need to chunk database schemas and table data. Passing raw CSV rows into an LLM often causes column-alignment confusion.

flowchart LR
    RawCSV["Raw CSV / Relational Rows<br/>id,name,role,dept<br/>101,Alice,Data Eng,Analytics"] --> Formatter["Markdown Table / Key-Value Formatter"]
    Formatter --> MD["Markdown Format<br/>| id | name | role | dept |<br/>| 101 | Alice | Data Eng | Analytics |"]
    Formatter --> KV["Key-Value Representation<br/>Record 1: id=101, name=Alice, role=Data Eng"]

    style RawCSV fill:#475569,color:#fff
    style Formatter fill:#2563eb,color:#fff
    style MD fill:#059669,color:#fff
    style KV fill:#7c3aed,color:#fff

Best Practice for Table Chunking:

1. Never split a table horizontally without preserving the header: Every single chunk of a 5,000-row table must retain the top column header line.
2. Key-Value String Representation: For sparse tables, convert each row into: "Customer ID: 101 | Region: US-West | Monthly Spend: $4,500 | Tier: Enterprise" Vector embedding models understand natural key-value sentences much better than raw commas.

3. Hands-on: Building a Recursive Text Splitter

Here is a clean Python implementation of recursive chunking with configurable overlap:

from typing import List

class RecursiveTextSplitter:
    def __init__(self, chunk_size: int = 200, chunk_overlap: int = 40):
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap
        self.separators = ["\n\n", "\n", " ", ""]

    def split_text(self, text: str) -> List[str]:
        return self._split(text, self.separators)

    def _split(self, text: str, separators: List[str]) -> List[str]:
        final_chunks = []
        separator = separators[-1]
        new_separators = []

        for i, sep in enumerate(separators):
            if sep == "":
                separator = ""
                break
            if sep in text:
                separator = sep
                new_separators = separators[i + 1:]
                break

        splits = text.split(separator) if separator != "" else list(text)
        good_splits = []

        for s in splits:
            if len(s) < self.chunk_size:
                good_splits.append(s)
            else:
                if good_splits:
                    final_chunks.extend(self._merge_splits(good_splits, separator))
                    good_splits = []
                if not new_separators:
                    final_chunks.append(s[:self.chunk_size])
                else:
                    final_chunks.extend(self._split(s, new_separators))

        if good_splits:
            final_chunks.extend(self._merge_splits(good_splits, separator))

        return final_chunks

    def _merge_splits(self, splits: List[str], separator: str) -> List[str]:
        chunks = []
        current_chunk = []
        current_length = 0

        for s in splits:
            s_len = len(s) + len(separator)
            if current_length + s_len > self.chunk_size:
                if current_chunk:
                    chunks.append(separator.join(current_chunk))
                    # Retain overlap from end of current chunk
                    while current_length > self.chunk_overlap and current_chunk:
                        popped = current_chunk.pop(0)
                        current_length -= (len(popped) + len(separator))
            current_chunk.append(s)
            current_length += s_len

        if current_chunk:
            chunks.append(separator.join(current_chunk))
        return chunks

if __name__ == "__main__":
    sample_doc = """
Apache Spark is a unified analytics engine for large-scale data processing.
It provides high-level APIs in Java, Scala, Python, and R.

Spark SQL is a Spark module for structured data processing.
It provides a programming abstraction called DataFrames and can act as a distributed SQL query engine.

Structured Streaming is a scalable stream processing engine built on Spark SQL.
You can express streaming computations the same way as batch computations.
""".strip()

    splitter = RecursiveTextSplitter(chunk_size=150, chunk_overlap=30)
    chunks = splitter.split_text(sample_doc)

    print(f"Generated {len(chunks)} Chunks:")
    for i, c in enumerate(chunks, 1):
        print(f"\n--- Chunk {i} ({len(c)} chars) ---")
        print(c.strip())

Key Takeaways

• Chunk size is a trade-off: 300 to 800 tokens is typically optimal for technical RAG applications.
• Always use chunk overlap (10% to 20%) to prevent context severing across paragraph boundaries.
• Recursive character chunking respects natural grammatical boundaries (`

$ o$ ` $ o$ spaces).

• For structured schemas and tables, include column headers in every chunk or convert rows into key-value strings.