Tokens and Embeddings: The Bridge Between Text and Math

Neural networks cannot process raw characters or strings. Before an LLM can reason, analyze, or generate words, text must be translated into numbers.

This translation happens in two distinct stages:

1. Tokenization: Converting strings into discrete integer IDs.
2. Embedding: Mapping discrete integer IDs into continuous, high-dimensional dense vector spaces.

Understanding this bridge is essential for data engineers: token counts dictate API bills, context window limits, and throughput in ETL pipelines.

1. What Is a Token?

A token is the atomic unit of text processed by an LLM. A token is neither strictly a character nor strictly a word:

• In English text, 1 token $\approx$ 4 characters or roughly 0.75 words.
• Common words (e.g., the, SELECT, database) are single tokens.
• Rare or compound words are split into subword fragments (e.g., PySpark $\to$ Py + Spark).
• Punctuation and spaces are often part of adjacent tokens.

flowchart LR
    Text["'SELECT count(*) FROM parquet_table;'"] --> Tokenizer["Subword Tokenizer<br/>(Byte-Pair Encoding)"]
    Tokenizer --> TokenIDs["[13840, 2487, 8569, 1423, 7812, 1182, 26]"]
    TokenIDs --> EmbeddingLayer["Embedding Matrix Lookup<br/>(Vocab Size × Dimension)"]
    EmbeddingLayer --> DenseVectors["Dense Tensor<br/>Shape: (7, 4096)"]

    style Text fill:#64748b,color:#fff
    style Tokenizer fill:#2563eb,color:#fff
    style TokenIDs fill:#7c3aed,color:#fff
    style EmbeddingLayer fill:#059669,color:#fff
    style DenseVectors fill:#d97706,color:#fff

The Subword Tokenization Breakthrough: Byte-Pair Encoding (BPE)

Why not just split by spaces (words) or split by letters (characters)?

• Word-level: Vocabulary explodes to millions of words, cannot handle misspellings or new jargon.
• Character-level: Sequences become 5x longer, destroying context windows and compute efficiency.
• Subword BPE (The Sweet Spot): Iteratively merges the most frequent pairs of bytes/characters until a fixed vocabulary size (e.g., 32,000 for LLaMA, 100,000 for GPT-4) is reached.

2. Token Economics for Data Engineers

When running LLM nodes in data pipelines (e.g., analyzing 500,000 customer feedback records daily), token efficiency directly drives cost and latency:

3. What Are Vector Embeddings?

Once text is tokenized into token IDs, an Embedding Layer maps each integer into a continuous vector space:

$$\mathbf{e}_i = \text{EmbeddingLookup}(w_i) \in \mathbb{R}^d$$

Where $d$ is the embedding dimension (e.g., 768 for small models, 4096 for LLaMA-3-70B, 1536 for OpenAI text-embedding-3-small).

The Magic of Semantic Vector Space

In this high-dimensional geometry:

• Similar concepts cluster together: The vector for "PostgreSQL" is closer to "MySQL" than to "Banana".
• Directional relationships encode meaning: $$\vec{v}(\text{"King"}) - \vec{v}(\text{"Man"}) + \vec{v}(\text{"Woman"}) \approx \vec{v}(\text{"Queen"})$$ $$\vec{v}(\text{"PySpark"}) - \vec{v}(\text{"Python"}) + \vec{v}(\text{"Scala"}) \approx \vec{v}(\text{"Spark Scala"})$$

quadrantChart
    title 2D Semantic Projection of Data Concepts
    x-axis "Analytical / Batch" --> "Transactional / Realtime"
    y-axis "Unstructured / GenAI" --> "Structured / Relational"
    quadrant-1 "OLTP Databases"
    quadrant-2 "Modern Warehouses"
    quadrant-3 "Vector Stores & RAG"
    quadrant-4 "Streaming & Queues"
    "PostgreSQL": [0.8, 0.85]
    "MySQL": [0.75, 0.8]
    "Snowflake": [0.25, 0.75]
    "BigQuery": [0.2, 0.7]
    "ChromaDB": [0.2, 0.2]
    "Pinecone": [0.35, 0.25]
    "Kafka": [0.85, 0.35]
    "Spark Streaming": [0.4, 0.4]

4. Measuring Vector Similarity

How do computers determine if two database queries or text documents mean the same thing? By comparing their vectors using distance metrics:

1. Cosine Similarity (The Industry Standard)

Measures the cosine of the angle between two vectors, regardless of their magnitude (normalized between -1.0 and +1.0):

$$\text{CosineSim}(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}| |\mathbf{v}|} = \frac{\sum_{i=1}^d u_i v_i}{\sqrt{\sum_{i=1}^d u_i^2} \sqrt{\sum_{i=1}^d v_i^2}}$$

• 1.0: Identical semantic meaning.
• 0.0: Orthogonal (completely unrelated).
• -1.0: Diametrically opposite meaning.

2. Dot Product

$$\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^d u_i v_i$$ If vectors are unit-normalized ($|\mathbf{u}| = 1$), dot product equals cosine similarity and is computationally faster to execute on GPUs and vector databases.

3. Euclidean Distance ($L_2$ Distance)

$$|\mathbf{u} - \mathbf{v}|2 = \sqrt{\sum{i=1}^d (u_i - v_i)^2}$$ Measures the straight-line distance between two points in space. Smaller distance indicates higher similarity.

5. Hands-on: Token Counting and Embedding Similarity

Here is a practical script demonstrating token analysis and semantic distance computation using standard Python tools:

import math
import numpy as np

def cosine_similarity(v1, v2):
    """Compute cosine similarity between two 1D NumPy arrays."""
    dot_prod = np.dot(v1, v2)
    norm_v1 = np.linalg.norm(v1)
    norm_v2 = np.linalg.norm(v2)
    if norm_v1 == 0 or norm_v2 == 0:
        return 0.0
    return float(dot_prod / (norm_v1 * norm_v2))

def simulate_embedding(text: str, dim: int = 128) -> np.ndarray:
    """
    Deterministic mock embedding generator for local demonstration without API keys.
    In production, replace with OpenAI, Cohere, or sentence-transformers embeddings.
    """
    np.random.seed(abs(hash(text.lower().strip())) % (2**32))
    vec = np.random.randn(dim)
    # Normalize to unit length
    return vec / np.linalg.norm(vec)

# Data Engineering Schema Semantic Matching Example
if __name__ == "__main__":
    queries = [
        "SELECT customer_id, SUM(order_total) FROM orders GROUP BY customer_id",
        "SELECT client_id, SUM(sales_amount) FROM transactions GROUP BY client_id",
        "DROP TABLE users CASCADE",
        "How do I bake sourdough bread with whole wheat flour?"
    ]

    print("--- Embedding Similarity Demonstration ---")
    vec_base = simulate_embedding(queries[0])

    for i, q in enumerate(queries):
        vec_q = simulate_embedding(q)
        sim = cosine_similarity(vec_base, vec_q)
        print(f"\nQuery [{i}]: \"{q}\"")
        print(f"Similarity with Query [0]: {sim:.4f}")

Key Takeaways

• Tokens are the currency of LLMs: Subword tokenization (BPE) balances vocabulary size with sequence length.
• Token counting is vital for data engineers to estimate pipeline costs, compute throughput, and avoid context window overflows.
• Embeddings map symbolic words into continuous geometrical coordinates where semantic meaning corresponds to spatial proximity.
• Cosine similarity is the mathematical foundation for vector search, Retrieval-Augmented Generation (RAG), and semantic schema matching.