llama.cpp and the Quantization You Actually Need
Q4, Q5, Q8 — the difference between a model that fits your GPU and one that does not, explained without the maths.
Quantization is the single most practical thing to understand about running models yourself, and almost every explanation of it starts with matrix mathematics nobody needs.
Here is the version that actually informs a decision.
The idea in one paragraph
A model stores billions of numbers. By default each one uses 16 bits. Quantization stores them with fewer bits — 8, 5, 4, sometimes 2 — which makes the file smaller and inference faster, at the cost of some precision. llama.cpp popularised the formats everyone now uses, and its GGUF files are what most local runners load underneath.
What the labels mean in practice
- Q8 — nearly indistinguishable from full precision. Use when you have memory to spare and quality is the priority.
- Q5_K_M — the sweet spot for most work. Noticeably smaller, quality loss you have to look for.
- Q4_K_M — the default I reach for. Roughly a quarter of the original size, still coherent, fits on consumer hardware.
- Q3 and below — visible degradation. Repetition, dropped instructions, worse non-English output. Only when nothing else fits.
Rough sizing: a 7B model at Q4 lands around 4 GB, at Q8 around 7 GB. Add a bit for context. That arithmetic tells you what your laptop can run before you download anything.
The part people miss
Quantization damage is not evenly distributed. It hits the edges of a model's competence hardest — and non-English output lives at that edge.
I have watched a model produce perfectly good English at Q3 while its Arabic degraded into grammatical mush. If you are building for Arabic, test the Arabic specifically at the quantization you plan to ship. The English benchmark everyone quotes will not warn you.
Long-context behaviour degrades similarly. A model that follows instructions well in a short prompt may start ignoring them at 8000 tokens once heavily quantized.
How I choose
Start at Q4_K_M. Run your actual prompts — not benchmarks, your prompts. If quality is short, step up to Q5 before considering a bigger model, because a larger model at Q3 is usually worse than a smaller one at Q5.
Resources
- Repo: ggml-org/llama.cpp
- Quantization notes in the repo's discussions and README
- Video walkthroughs: YouTube: llama.cpp quantization
- Easiest way to consume it: Ollama
Need this built properly?
I build secure, fast, bilingual platforms for clients across Egypt, Saudi Arabia, the UAE and Kuwait.
