Chapter 2: Understanding Foundation Models

+-------------------------------------------------------------------------------+
|                        AUTOREGRESSIVE NEXT-TOKEN PREDICTION                   |
+-------------------------------------------------------------------------------+
|  Input Tokens: ["The", " modern", " data", " stack"]                          |
|  Forward Pass: Compute probability distribution over vocabulary V (~128k)     |
|  Sampling:     Argmax / Nucleus (top-p) -> Next Token: [" relies"]            |
|  Append Token: Re-feed sequence back into Context Window                      |
+-------------------------------------------------------------------------------+

Core Mechanics of Modern LLMs

Foundation models are primarily decoder-only transformers trained on trillions of tokens via causal self-attention.

1. Tokenization Realities

• Models do not see characters or words; they see token IDs.
• Byte-Pair Encoding (BPE) chunks frequent character sequences into integers.
• Data Engineering Impact: Token counts dictate API bills, context window exhaustion, and latency. Numbers and spaces often split into multiple arbitrary tokens.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Snowflake & Databricks 2026")
print(f"Token count: {len(tokens)}")
print(f"Token IDs: {tokens}")
print(f"Deconstructed: {[enc.decode([t]) for t in tokens]}")

Scaling Laws & Emergence

Chip highlights the Chinchilla scaling laws (Hoffmann et al.):

• For compute-optimal training, model parameter count and training token volume should scale in equal proportion.
• Most modern models (Llama 3, Mistral) are deliberately overtrained (trained on 15T+ tokens, far beyond the Chinchilla minimum) to minimize serving inference costs down the road.

Context Window vs. Effective Context

While context windows have exploded to 128k, 1M, or 2M tokens:

• Needle-in-a-Haystack problem: Recall accuracy degrades in the middle of long documents ("Lost in the Middle").
• Quadratic Cost: Standard full attention computes $O(N^2)$ dot products where $N$ is sequence length.
• Key Takeaway: Even with a 1M context window, high-precision RAG (retrieving the top 3 targeted chunks) consistently beats dumping 500 pages into a prompt.