RLHF and DPO (Direct Preference Optimization)

+-----------------------------------------------------------------------------------+
|                        PPO (RLHF) VS. DPO ARCHITECTURE                            |
+-----------------------------------------------------------------------------------+
|  Traditional RLHF (PPO):                                                          |
|  SFT Model -> Train Reward Model (RM) -> Run Reinforcement Learning (PPO) Actor   |
|  (Complex, unstable, requires 4 separate models in GPU memory!)                  |
|                                                                                   |
|  Direct Preference Optimization (DPO):                                            |
|  Directly optimizes the LLM policy on pairwise preference data: (Prompt, Y_w, Y_l)|
|  Y_w = Chosen (Winning) response                                                  |
|  Y_l = Rejected (Losing) response                                                 |
|  (Stable, closed-form loss, no separate reward model needed!)                     |
+-----------------------------------------------------------------------------------+

Why DPO Replaced PPO for Most Teams

Proximal Policy Optimization (PPO) was used to train InstructGPT and ChatGPT. However, PPO is notorious for:

• Hyperparameter sensitivity (policy collapses easily).
• High GPU memory overhead (running the Actor, Critic, Reference, and Reward models simultaneously).

In 2023, Rafael Rafailov et al. published Direct Preference Optimization (DPO), mathematically proving that the reinforcement learning objective can be derived directly from the classification loss of chosen vs. rejected outputs:

$$\mathcal{L}{ ext{DPO}}(\pi heta; \pi_{ ext{ref}}) = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( eta \log rac{\pi heta(y_w|x)}{\pi_{ ext{ref}}(y_w|x)} - eta \log rac{\pi_ heta(y_l|x)}{\pi_{ ext{ref}}(y_l|x)} ight) ight]$$

DPO Dataset Format

{
  "prompt": "Write a query to calculate month-over-month revenue growth.",
  "chosen": "SELECT month, revenue, (revenue - LAG(revenue) OVER (ORDER BY month)) / LAG(revenue) OVER (ORDER BY month) AS mom_growth FROM sales;",
  "rejected": "SELECT month, (revenue - previous_revenue) / 100 FROM sales;"
}

Running DPO with Hugging Face TRL

from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

# Configuration
training_args = DPOConfig(
    learning_rate=5e-6,
    beta=0.1, # Implicit reward scaling parameter
    per_device_train_batch_size=2,
    output_dir="./dpo_results",
)

# dpo trainer = DPOTrainer(
#     model=model,
#     ref model=None, # TRL automatically creates a reference copy if None
#     args=training args,
#     train dataset=dataset,
#     tokenizer=tokenizer,
# )
# dpo trainer.train()