Theoretical Quiz: AI Engineering (Chip Huyen)
Question 1
According to Chip Huyen's AI Engineering methodology, why is token generation during the decoding phase memory-bound rather than compute-bound?
- A) Because the GPU cores shut down during generation.
- B) Because for every single generated token, the entire model parameter weight matrix must be read from GPU High Bandwidth Memory (HBM) into processing cores.
- C) Because text tokens take up more hard drive space than images.
- D) Because Python strings are slow.
Answer: B
Explanation: In autoregressive generation, generating 1 token requires loading all model weights from HBM to SRAM, making memory bandwidth the primary bottleneck.
Question 2
What is the primary difference between Dense Retrieval and Sparse Retrieval in modern RAG architectures?
- A) Dense retrieval runs on CPU; sparse runs on GPU.
- B) Dense retrieval captures semantic and conceptual meaning via embedding vectors, while sparse retrieval (BM25) captures exact keywords, IDs, acronyms, and rare tokens.
- C) Dense retrieval is free; sparse retrieval is paid.
- D) Dense retrieval requires fine-tuning; sparse does not.
Answer: B
Explanation: Combining both with Reciprocal Rank Fusion provides the highest accuracy because they complement each other's weaknesses.
Question 3
What is the "Needle-in-a-Haystack" phenomenon in long-context foundation models?
- A) Models run out of memory when reading more than 10 pages.
- B) Model retrieval accuracy drops significantly when crucial facts are located in the middle of a very long prompt context ("Lost in the Middle").
- C) Models refuse to answer financial questions.
- D) The tokenizer fails on special punctuation characters.
Answer: B
Explanation: Even when models support 128k+ context windows, attention distributions attenuate in the middle of long documents, often causing facts placed there to be overlooked.