Ollama: Run Real Models on Your Own Machine

The first time a client asked whether their customer data would be sent to a third party, I did not have a good answer. Now I do, and it takes one command.

Ollama pulls a model, quantizes it sensibly, and serves it on localhost:11434 with an API close enough to what you already use that swapping is a config change.

What it actually does

It is a model runner with a package manager attached. You do not think about weights, formats, or GPU offload layers:

ollama pull qwen2.5:7b
ollama run qwen2.5:7b "summarise this contract clause"

The same model is then available over HTTP, which is the part that matters for building anything real:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "Extract the delivery date",
  "stream": false
}'

How I use it

Prototyping without a meter running. Early in a feature I iterate on prompts dozens of times an hour. Doing that against a paid API is a slow way to spend money on experiments that get thrown away.

Data that cannot leave. For a client handling personal records, "the text never leaves this server" ends the compliance conversation before it starts.

Offline demos. Conference wifi has ruined enough presentations. A local model does not care.

A fallback path. When a provider has an outage, degrading to a smaller local model beats showing an error.

The catch

A 7B model is not a frontier model, and pretending otherwise will embarrass you in front of a client. For summarisation, extraction, classification and routing, the gap is small. For complex reasoning across a long document, it is not.

The honest workflow is to develop locally and evaluate both — then pick per task rather than per project. Plenty of features work fine on a small local model, and the ones that do not are usually obvious within an afternoon of testing.

Also budget the hardware conversation early. A 7B model at 4-bit needs roughly 5–6 GB of memory to be comfortable; anything larger means a real GPU or a very patient user.

Resources

• Repo: ollama/ollama
• Model library: ollama.com/library
• Video walkthroughs: YouTube: ollama tutorial
• Pairs well with Open WebUI for a client-facing interface