Chapter 9: Inference Optimization

+---------------------------------------------------------------------------------+
|                       WHERE DOES INFERENCE TIME GO?                             |
+---------------------------------------------------------------------------------+
|  Phase 1: Prefill (Prompt Processing) -> Compute-Bound (Processes all tokens)   |
|  Phase 2: Decoding (Token Generation) -> Memory-Bound (Reads weights every step)|
+---------------------------------------------------------------------------------+

The Memory Bandwidth Bottleneck

During the autoregressive decoding phase, an LLM must read every parameter in the model from GPU HBM (High Bandwidth Memory) into the tensor processing cores for each single generated token.

This means token generation speed is bottlenecked by memory bandwidth, not raw compute FLOPs!

Modern Inference Acceleration Techniques

1. KV Caching & PagedAttention (vLLM)

• The Key-Value (KV) cache stores past token keys and values so attention doesn't recompute history on every step.
• Traditional KV caches caused huge memory fragmentation.
• PagedAttention (developed by Woosuk Kwon et al.) divides KV caches into virtual memory pages, enabling 2x to 4x higher concurrency on the same GPU.

2. Quantization (AWQ, GPTQ, INT4)

• Reduces weight size from 16 bits to 4 bits.
• Less bytes transferred over memory bus $\implies$ faster token decoding speed.

3. Speculative Decoding

• A tiny, fast "draft model" (e.g. 1B parameter model) generates 5 candidate tokens speculatively in parallel.
• The large 70B model verifies all 5 tokens in a single fast forward pass.
• Yields 2x to 3x speedups with mathematically identical output distributions.