Module 1 Theoretical & Architecture Interview Quiz
10 high-impact technical interview questions for Data Engineers and AI Engineers transitioning into Generative AI roles.
Q1: Why is Self-Attention computationally $O(N^2)$ with respect to sequence length, and what breakthrough solved this for modern long-context models?
👉 Reveal Answer & Architectural Deep Dive
**Answer**: 1. **The $O(N^2)$ Reason**: The core equation of self-attention computes $\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)$. Since $Q$ has dimensions $(N \times d_k)$ and $K^T$ has dimensions $(d_k \times N)$, the matrix product $QK^T$ produces an $N \times N$ affinity matrix. Every single token must compute a dot product against every other token in the sequence. Doubling context length from 4k to 8k quadruples both the memory allocations and FLOP requirements. 2. **The Breakthrough**: **FlashAttention** (Dao et al.). Standard attention materializes the intermediate $N \times N$ matrix in high-latency GPU High-Bandwidth Memory (HBM). FlashAttention breaks the inputs into blocks, loads them into fast on-chip GPU SRAM, and computes attention using online softmax normalization, never writing the massive $N \times N$ matrix back to memory. This eliminates the memory bottleneck and reduces wall-clock execution time by 2x to 4x.Q2: What is the Key-Value (KV) Cache, and why does it represent a major bottleneck for LLM serving throughput?
👉 Reveal Answer & Architectural Deep Dive
**Answer**: During inference, LLMs generate tokens autoregressively (one token at a time). To predict token $t+1$, the model requires keys and values from all prior tokens $1 \dots t$. Instead of re-evaluating all prior tokens through 80+ transformer layers on every single step ($O(N^3)$ computational disaster), the Key and Value tensors are saved in GPU VRAM (the KV-Cache). **The Bottleneck**: The KV-Cache grows linearly with both **batch size** and **sequence length**: $$\text{Memory} = 2 \times \text{Layers} \times \text{Heads} \times \text{HeadDim} \times \text{Tokens} \times \text{BatchSize}$$ Under concurrent enterprise load (e.g., 50 parallel users generating long summaries), the KV-cache can easily consume 40+ GB of GPU memory, leading to out-of-memory (OOM) errors even when the model weights themselves fit comfortably.Q3: Explain the difference between Byte-Pair Encoding (BPE) and WordPiece tokenization.
👉 Reveal Answer & Architectural Deep Dive
**Answer**: Both are **subword tokenization algorithms** that iteratively merge characters into frequent chunks: - **Byte-Pair Encoding (BPE)**: Counts the absolute frequency of adjacent token pairs across the corpus and greedily merges the most frequent pair. Used by GPT-2, GPT-4, LLaMA, and RoBERTa. Modern implementations operate at the raw **byte** level so any unicode character can be represented without generating `Q4: According to the Chinchilla scaling law, what was the primary flaw in models like GPT-3 (175B parameters) and Gopher (280B parameters)?
👉 Reveal Answer & Architectural Deep Dive
**Answer**: They were severely **undertrained** relative to their parameter size. DeepMind's Chinchilla paper (2022) proved that for compute-optimal training, parameter count and training tokens should scale in equal proportion ($D \approx 20 \times N$). - GPT-3 had 175 billion parameters but was only trained on 300 billion tokens ($pprox 1.7 \times N$). - Chinchilla showed that a smaller **70-billion parameter model** trained on **1.4 trillion tokens** ($20 \times N$) outperformed GPT-3 175B across almost every benchmark, while being much cheaper and faster to serve at inference time.Q5: In an enterprise ETL pipeline, why would you choose an Encoder-Only model (like BERT or ModernBERT) over a Decoder-Only model (like GPT-4o)?
👉 Reveal Answer & Architectural Deep Dive
**Answer**: 1. **Bidirectional Context**: Encoder-only models attend to both left and right context simultaneously, making them superior for producing compact, dense **sentence embeddings** and token-level entity extraction (NER). 2. **Cost & Latency**: Running BERT/ModernBERT locally on CPU or a micro-GPU costs pennies and runs with sub-10ms latency, compared to 500ms+ network latency and recurring API fees for frontier decoder models. 3. **Determinism**: Encoders produce deterministic numerical vectors without risk of hallucinations or formatting drift.Q6: What is the "Lost in the Middle" phenomenon, and how should a data engineer architect prompts to counter it?
👉 Reveal Answer & Architectural Deep Dive
**Answer**: **Phenomenon**: Language models retrieve and reason over facts positioned at the **beginning (primacy effect)** and **end (recency effect)** of the context window with significantly higher fidelity than facts positioned in the middle 50% of the context. **Engineering Mitigations**: 1. Place strict schemas, formatting requirements, and critical validation rules in the **System Prompt** (beginning). 2. Place the immediate analytical question or input payload at the **very end** of the prompt. 3. In RAG pipelines, sort retrieved context chunks so that the most relevant chunks are placed at the beginning and the end of the context, rather than in the center.Q7: What is the difference between Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO)?
👉 Reveal Answer & Architectural Deep Dive
**Answer**: - **SFT (Supervised Fine-Tuning)**: Trains the model on curated target demonstrations using standard cross-entropy loss: $\mathcal{L} = -\sum \log P(y_t \mid x, y_{👉 Reveal Answer & Architectural Deep Dive
**Answer**: Quantization compresses model weights from 16-bit floating point (`BF16`/`FP16`, 2 bytes) to 4-bit integers (`INT4`, 0.5 bytes). - **Memory Impact**: A 70-billion parameter model requires ~140 GB of VRAM in FP16. In INT4, the model weights take only ~35 GB, allowing it to fit onto a single 48GB/80GB GPU. - **Throughput / Latency**: LLM inference is **memory-bandwidth bound**, not compute-bound. Because transferring 4-bit weights from GPU memory to tensor cores requires 75% less memory bus traffic, INT4 inference often runs **faster** than FP16 on memory-constrained hardware with <1% degradation in benchmark accuracy.👉 Reveal Answer & Architectural Deep Dive
**Answer**: Euclidean distance measures the magnitude of the difference vector: $\|u - v\|$. If document A is 100 words and document B is 5,000 words discussing the identical topic, document B's raw embedding vector may have a much larger magnitude, resulting in high Euclidean distance despite identical semantic topic. Cosine similarity evaluates only the **angle $\theta$** between vectors: $$\cos(\theta) = \frac{u \cdot v}{\|u\| \|v\|}$$ It normalizes by vector length, making similarity invariant to document length. Furthermore, if embeddings are pre-normalized to unit length ($\|u\|=1$), Cosine Similarity simplifies to a simple dot product $u \cdot v$, which vector databases can calculate at billions of vectors per second using AVX-512 or GPU tensor cores.👉 Reveal Answer & Architectural Deep Dive
**Answer**: | Factor | 1M Context Window | Vector DB + RAG | | :--- | :--- | :--- | | **Cost** | High ($0.50 – $2.00 per query) | Extremely Low (< $0.005 per query) | | **Latency** | 10 – 30 seconds Time-to-First-Token | Sub-second retrieval + generation | | **Data Updates** | Must resend the entire corpus on every change | Real-time incremental upserts to vector indices | | **Citation & Audit** | Hard to track which token influenced the generation | Exact document ID, page, and chunk metadata returned | | **Ideal Use Case** | Whole-codebase refactoring, financial book audits | Enterprise search, multi-tenant knowledge bases |
Find this content helpful?
☕ Buy me a coffee