Full Fine-Tuning vs. PEFT (Parameter-Efficient Fine-Tuning)

+---------------------------------------------------------------------------------+
|                       FULL FINE-TUNING VS. PEFT (LoRA)                          |
+---------------------------------------------------------------------------------+
|  Full Fine-Tuning:                                                              |
|  - Updates 100% of weights (e.g. 70 Billion parameters)                         |
|  - Requires 16x VRAM (weights + gradients + Adam optimizer states)               |
|  - Risk of Catastrophic Forgetting                                              |
|                                                                                 |
|  PEFT (LoRA / QLoRA):                                                           |
|  - Freezes 99.8% of base weights                                                |
|  - Trains tiny low-rank adapter matrices (0.2% of weights)                      |
|  - Runs on a single consumer GPU                                                |
|  - Zero catastrophic forgetting of foundational reasoning                      |
+---------------------------------------------------------------------------------+

The Hidden Cost of Optimizer States

Why can't you easily full-fine-tune an 8B model on a single 24GB GPU? In 16-bit precision, an 8B model's weights occupy 16 GB. But during standard AdamW optimization:

• Model Weights: 16 GB (FP16)
• Gradients: 16 GB (FP16)
• Optimizer State 1 (Momentum): 32 GB (FP32)
• Optimizer State 2 (Variance): 32 GB (FP32)
• Activations & Cache: ~16-24 GB

Total VRAM needed for Full Fine-Tuning: ~112 GB to 140 GB (requiring multi-GPU cluster like 2x A100 80GB).

The PEFT Revolution

Parameter-Efficient Fine-Tuning (PEFT) freezes the billions of pre-trained parameters and only trains a tiny fraction (< 1%) of auxiliary parameters:

• Prefix Tuning / Prompt Tuning: Appends trainable virtual token embeddings to keys/values.
• LoRA (Low-Rank Adaptation): Injects trainable rank-decomposition matrices into transformer attention layers.
• QLoRA: Combines 4-bit base model quantization with FP16/BF16 LoRA adapters.

graph LR
    Input[Input x] --> Base[Base Frozen Linear Layer W0]
    Input --> LoRA_A[Trainable Matrix A: d x r]
    LoRA_A --> LoRA_B[Trainable Matrix B: r x d]
    Base --> Add(( + ))
    LoRA_B --> Scale[Scale: alpha / r]
    Scale --> Add
    Add --> Output[Output y]