LoRA and QLoRA Deep Dive

+-----------------------------------------------------------------------------------+
|                            THE MATHEMATICS OF LoRA                                |
+-----------------------------------------------------------------------------------+
|  Original weight matrix: W0 in R^(d x k)                                          |
|  Instead of updating full delta: W = W0 + Delta W                                 |
|  Factorize Delta W into two low-rank matrices: Delta W = B * A                    |
|  where A in R^(r x k) and B in R^(d x r), with rank r << min(d, k)                |
|                                                                                   |
|  Example: d = 4096, k = 4096.                                                     |
|  Full Delta W parameters: 4096 * 4096 = 16,777,216                                |
|  LoRA with rank r = 16: (4096 * 16) + (16 * 4096) = 131,072  (>99.2% reduction!) |
+-----------------------------------------------------------------------------------+

How LoRA Works

During inference or forward pass, the input $x$ is passed through both the frozen original weight $W_0$ and the adapter branch:

$$h = W_0 x + \Delta W x = W_0 x + rac{lpha}{r} (B A) x$$

• Initialization: Matrix $A$ is initialized with random Gaussian values, and Matrix $B$ is initialized to zero. Thus, at step 0, $\Delta W = 0$, meaning the model begins with the exact baseline behavior!
• Scaling Factor $rac{lpha}{r}$: $lpha$ (alpha) is a constant hyperparameter that scales the adapter's influence. Typically, $lpha$ is set to $2 imes r$ or $1 imes r$.

What Is QLoRA (Quantized LoRA)?

QLoRA (introduced by Tim Dettmers et al.) made it possible to fine-tune a 70B parameter model on a single 48GB GPU (or an 8B model on a single 16GB GPU) via three innovations:

1. 4-bit NormalFloat (NF4): An information-theoretically optimal quantile quantization data type for normally distributed neural network weights.
2. Double Quantization (DQ): Quantizing the quantization constants themselves, saving an extra 0.37 bits per parameter.
3. Paged Optimizers: Using CUDA unified memory to automatically page optimizer states to CPU RAM during activation memory spikes, preventing Out-Of-Memory (OOM) crashes.

Selecting Target Modules

In modern Transformer architectures (Llama, Mistral, Qwen), LoRA can be applied to:

• Attention projections: q_proj, k_proj, v_proj, o_proj
• Feed-Forward / MLP layers: gate_proj, up_proj, down_proj

Targeting all linear layers yields the highest task adaptation performance while still keeping trainable parameters under 1-2% of the base model.