Theoretical Quiz: Fine-Tuning & RLHF
Question 1
Why does initializing LoRA Matrix $B$ to zero and Matrix $A$ to Gaussian random values guarantee safe training kickoff?
- A) It prevents gradient vanishing.
- B) Because $\Delta W = B \cdot A$, initializing $B=0$ ensures that $\Delta W = 0$ at step 0, meaning the model starts with the exact behavior and capabilities of the pre-trained base model.
- C) It guarantees 100% test accuracy on epoch 1.
- D) It quantizes the network automatically.
Answer: B
Explanation: If $B=0$, the adapter initially contributes nothing to the output ($W = W_0 + 0$), ensuring no disruption to baseline knowledge.
Question 2
What is the primary advantage of Direct Preference Optimization (DPO) over Proximal Policy Optimization (PPO)?
- A) DPO requires no training data whatsoever.
- B) DPO eliminates the need to train a separate reward model and avoids running unstable RL loops, optimizing the policy directly with an exact closed-form objective on pairwise preference data.
- C) DPO only works on proprietary closed models.
- D) DPO runs 100% on CPU without GPU requirements.
Answer: B
Explanation: DPO mathematically re-parameterizes the reward function in terms of the policy, allowing direct optimization without an auxiliary reward model or PPO complexity.
Question 3
Under what condition should a Data Engineer choose RAG instead of Fine-Tuning?
- A) When teaching an LLM to generate a proprietary SQL dialect.
- B) When the underlying data changes frequently (e.g., daily sales, updated warehouse documentation, new schemas) and requires verifiable source citations.
- C) When trying to minimize token consumption.
- D) When the model needs to run completely offline on a smartphone.
Answer: B
Explanation: Fine-tuning is unsuitable for dynamic, rapidly changing facts. RAG retrieves fresh facts at query time with direct attribution.