How LLMs Are Trained: From Trillions of Tokens to Intelligence

Training a modern foundation model is one of the largest engineering feats in human history. It requires orchestrating thousands of specialized GPUs across ultra-low-latency InfiniBand clusters, executing continuous distributed matrix multiplications for months without failure.

For data engineers, understanding this training lifecycle is crucial: over 70% of modern LLM improvements come from data engineering (curation, deduplication, filtering, and synthetic generation), not algorithmic changes.

1. The 3-Stage Training Pipeline

flowchart TD
    subgraph Stage1["Stage 1: Pre-Training (90-95% of Compute)"]
        Raw["Raw Web Scraping, Books, Code, Papers<br/>(15+ Trillion Tokens)"]
        DE["PySpark Data Engineering Pipeline<br/>(Deduplication, Quality Filtering, PII Redaction)"]
        Cluster["GPU Supercluster (Thousands of H100s)<br/>Next-Token Prediction Objective"]
        Base["Foundation / Base Model<br/>(e.g., LLaMA-3-8B-Base)"]

        Raw --> DE --> Cluster --> Base
    end

    subgraph Stage2["Stage 2: Supervised Fine-Tuning SFT"]
        InstructData["Curated High-Quality Pairs<br/>(100k - 1M Instructions & Expert Answers)"]
        InstructTune["Instruction Tuning Training Loop"]
        InstructModel["Instruct-Tuned Model<br/>(Conversational & Task Follower)"]

        Base --> InstructTune
        InstructData --> InstructTune
        InstructTune --> InstructModel
    end

    subgraph Stage3["Stage 3: Alignment (RLHF / DPO)"]
        Feedback["Human & AI Feedback Pairings<br/>(Chosen vs Rejected Completions)"]
        DPO["Direct Preference Optimization / PPO"]
        ProdModel["Production Model<br/>(Safe, Helpful, Calibrated)"]

        InstructModel --> DPO
        Feedback --> DPO
        DPO --> ProdModel
    end

    style Raw fill:#334155,color:#fff
    style DE fill:#2563eb,color:#fff
    style Base fill:#059669,color:#fff
    style InstructModel fill:#7c3aed,color:#fff
    style ProdModel fill:#d97706,color:#fff

2. The Data Engineering Behind Pre-Training

Feeding messy, uncurated data into a model wastes millions of dollars in compute. Modern pre-training pipelines filter out over 80% of raw internet scrapes using rigorous data engineering stages:

flowchart LR
    A["Raw Common Crawl<br/>100 TBs"] -->|URL & Domain Filtering| B["Filtered URLs<br/>50 TBs"]
    B -->|Language ID & Quality Heuristics| C["High-Quality Text<br/>25 TBs"]
    C -->|MinHash LSH Deduplication| D["De-duplicated Corpus<br/>10 TBs"]
    D -->|PII Redaction & Safety| E["Final Pre-training Set<br/>~15 Trillion Tokens"]

    style A fill:#475569,color:#fff
    style B fill:#3b82f6,color:#fff
    style C fill:#06b6d4,color:#fff
    style D fill:#8b5cf6,color:#fff
    style E fill:#10b981,color:#fff

Critical Data Cleaning Tasks Handled by Data Engineers:

1. MinHash Locality-Sensitive Hashing (LSH): Detects near-duplicate documents across petabytes of text. Repeated text causes LLMs to degenerate into repetitive loops.
2. Quality Heuristics: Filtering out documents with low punctuation ratios, excessive repetitive n-grams, or high boilerplate-to-content ratios.
3. PII Redaction: Scanning and masking Social Security Numbers, emails, phone numbers, and credentials before tokenization.

3. Scaling Laws: Parameters vs. Tokens

How large should a model be, and how many tokens should it read?

In 2022, DeepMind published the seminal Chinchilla Scaling Paper (Hoffmann et al.), proving that most contemporary models were severely undertrained:

4. Hardware, VRAM, and Quantization

Understanding memory requirements is essential when deploying LLMs on AWS EC2, EKS, or serverless containers:

Model Precision Formats

• FP32 (32-bit Float): 4 bytes per parameter. Full precision (rarely used for LLMs).
• FP16 / BF16 (16-bit Float): 2 bytes per parameter. The universal standard for training and high-accuracy inference.
• INT8 (8-bit Quantized): 1 byte per parameter. 50% memory reduction with negligible (<1%) loss in quality.
• INT4 (4-bit Quantized - AWQ/GPTQ/GGUF): 0.5 bytes per parameter. Allows 7B models to run on mobile phones or low-cost 8GB GPUs!

Inference VRAM Calculation Rule of Thumb:

$$\text{VRAM Required} \approx \text{Parameters (in Billions)} \times \text{Bytes per Parameter} \times 1.25 \text{ (Overhead)}$$

5. Hands-on: GPU Memory Estimator Script

Use this utility to calculate exact memory requirements for hosting open-weight models on cloud infrastructure:

def estimate_llm_memory(
    param_count_billions: float, 
    precision_bits: int = 16, 
    context_tokens: int = 4096, 
    batch_size: int = 1
):
    """
    Calculate exact GPU VRAM requirement for serving an LLM model,
    including model weights and the KV-Cache.
    """
    bytes_per_param = precision_bits / 8.0
    weights_gb = (param_count_billions * 1e9 * bytes_per_param) / (1024**3)

    # KV Cache estimate: 2 * num_layers * hidden_size * num_heads * context_len
    # Heuristic: ~0.5MB per token per batch for a typical 7B model at FP16
    kv_cache_mb = (context_tokens * batch_size * (param_count_billions / 7.0) * 0.5)
    kv_cache_gb = kv_cache_mb / 1024.0

    # 20% CUDA runtime & memory fragmentation overhead
    total_gb = (weights_gb + kv_cache_gb) * 1.20

    return {
        "model_parameters": f"{param_count_billions}B",
        "precision": f"{precision_bits}-bit",
        "weights_memory_gb": round(weights_gb, 2),
        "kv_cache_gb": round(kv_cache_gb, 2),
        "total_recommended_vram_gb": round(total_gb, 2)
    }

if __name__ == "__main__":
    configs = [
        (8.0, 16, 8192),   # LLaMA-3-8B in FP16
        (8.0, 4, 8192),    # LLaMA-3-8B in INT4
        (70.0, 16, 8192),  # LLaMA-3-70B in FP16
        (70.0, 4, 8192),   # LLaMA-3-70B in INT4
    ]

    print("=== Production VRAM Deployment Sizing Guide ===")
    for params, bits, ctx in configs:
        res = estimate_llm_memory(params, bits, ctx)
        print(f"\nModel: {res['model_parameters']} ({res['precision']}) @ {ctx} context:")
        print(f"  - Weights Memory : {res['weights_memory_gb']} GB")
        print(f"  - KV Cache Memory: {res['kv_cache_gb']} GB")
        print(f"  - Total VRAM Need: {res['total_recommended_vram_gb']} GB")

Key Takeaways

• Pre-training is compute-intensive ($90\%+$$ of total cost) and turns raw tokens into general statistical world models.
• 70%+ of model quality comes from data engineering: Deduplication with MinHash LSH, PII masking, and quality filtering are the real secret weapons.
• SFT and DPO alignment mold raw token predictors into safe, helpful, instruction-following agents.
• Quantization (INT8/INT4) reduces VRAM footprint by up to 75%, allowing production models to run on affordable commodity cloud instances.