Hands-on Quiz: Hugging Face & Open-Source Models
Test your ability to interact with Hugging Face models and run local quantized inference.
Challenge 1: Memory Footprint Estimator
Write a Python function that computes the expected VRAM memory requirement (in Gigabytes) to run an open-source model given parameter count (in billions) and precision bits. Include a 20% overhead factor for KV cache and activations.
def estimate_vram_gb(param_count_billions: float, precision_bits: int) -> float:
# Your implementation here
pass
View Solution
def estimate_vram_gb(param_count_billions: float, precision_bits: int) -> float:
bytes_per_param = precision_bits / 8.0
raw_bytes = (param_count_billions * 1e9) * bytes_per_param
with_overhead = raw_bytes * 1.20
return round(with_overhead / (1024 ** 3), 2)
# Test: 8B model at 4-bit
print(estimate_vram_gb(8.0, 4)) # ~4.47 GB
Challenge 2: Streaming Ollama Completion
Write a Python script using the requests library that streams the generated tokens from a local Ollama instance line-by-line as they are produced.
View Solution
import json
import requests
def stream_ollama(prompt: str, model: str = "llama3.1"):
url = "http://localhost:11434/api/generate"
payload = {"model": model, "prompt": prompt, "stream": True}
with requests.post(url, json=payload, stream=True) as response:
for line in response.iter_lines():
if line:
chunk = json.loads(line.decode("utf-8"))
print(chunk.get("response", ""), end="", flush=True)
if chunk.get("done", False):
break
print()
Find this content helpful?
☕ Buy me a coffee