Quantization, GGUF, and Ollama

+-------------------------------------------------------------------------------+
|                            HOW QUANTIZATION WORKS                             |
+-------------------------------------------------------------------------------+
|  FP32 (32-bit float)  -> 4 bytes per weight   -> Huge memory, extreme precision|
|  FP16 (16-bit float)  -> 2 bytes per weight   -> Standard training precision  |
|  INT8 (8-bit integer) -> 1 byte per weight    -> 50% memory reduction, ~0 loss|
|  INT4 (4-bit integer) -> 0.5 bytes per weight -> 75% memory drop, runnable on  |
|                                                  MacBook / Edge hardware     |
+-------------------------------------------------------------------------------+

What Is Quantization?

Quantization maps continuous high-precision floating point numbers into discrete lower-precision representations. By quantizing weights from 16-bit to 4-bit (using algorithms like AWQ, GPTQ, or GGUF k-quants), an 8-billion parameter model drops from 16 GB down to ~4.8 GB with virtually no loss in conversational ability.

What is GGUF?

GGUF (GPT-Generated Unified Format) is the binary format developed by Georgi Gerganov for llama.cpp. A .gguf file stores:

• All model tensor weights in a single unified file.
• The tokenizer vocabulary.
• Quantization parameters and metadata.
• Allows instant memory-mapping (mmap) into system RAM without slow initialization.

Running Ollama Locally for Zero-Cost AI

Ollama wraps llama.cpp in an intuitive CLI and REST API daemon that auto-selects CPU, GPU, or Apple Silicon acceleration:

1. Installation & Running a Model

# Install Ollama (Mac/Linux/Windows)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run Llama 3.1 8B in 4-bit quantization
ollama run llama3.1

2. Calling Ollama with Python

import requests

def call_ollama(prompt: str, model: str = "llama3.1") -> str:
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model,
        "prompt": prompt,
        "stream": False,
        "options": {
            "temperature": 0.1
        }
    }
    response = requests.post(url, json=payload)
    return response.json()["response"]

print(call_ollama("Write an Airflow BashOperator that unzips a file."))