Running Open-Source LLMs Locally

+-------------------------------------------------------------------------+
|                  VRAM MEMORY REQUIREMENT FORMULA                        |
+-------------------------------------------------------------------------+
|  Memory (Bytes) = Parameters * (Bits per Parameter / 8) * 1.2 overhead  |
|                                                                         |
|  Example for 8 Billion Parameter Model:                                 |
|  - 16-bit (FP16): 8B * 2 bytes = 16 GB VRAM                             |
|  - 8-bit (INT8):  8B * 1 byte  = 8 GB VRAM                              |
|  - 4-bit (INT4):  8B * 0.5 byte = 4 GB VRAM                             |
+-------------------------------------------------------------------------+

Hardware Acceleration Options

Running open-source LLMs requires selecting the right accelerator device:

1. NVIDIA GPUs (CUDA): The gold standard for production inference and fine-tuning. Supported natively by PyTorch, vLLM, and TensorRT-LLM.
2. Apple Silicon (MPS & MLX): Apple unified memory allows Mac M-series chips (M1/M2/M3/M4) to allocate up to 128GB of RAM directly to the GPU without PCIe transfer bottlenecks.
3. CPU Inference (llama.cpp): Fast AVX-512 quantized inference on standard commodity x86 servers.

Running with PyTorch on CUDA or Apple Silicon (MPS)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

# Detect best available accelerator device
if torch.cuda.is_available():
    device = "cuda"
    dtype = torch.bfloat16
elif torch.backends.mps.is_available():
    device = "mps"
    dtype = torch.float16
else:
    device = "cpu"
    dtype = torch.float32

print(f"Running inference on: {device} with {dtype}")

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=dtype,
    device_map="auto"
)

prompt = "Explain Apache Spark shuffle in two sentences."
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=100)

print(tokenizer.decode(output[0], skip_special_tokens=True))