vLLM: When You Outgrow One Request at a Time
Continuous batching and paged attention are why a serving stack handles fifty concurrent users on hardware that choked on five.
A local model serving one developer is a toy. A local model serving a hundred concurrent users is infrastructure, and the gap between those two is not hardware — it is scheduling.
vLLM is what closes it.
The problem it solves
Naive serving handles one request at a time, or batches requests that arrive together and makes the fast ones wait for the slowest. GPU memory sits fragmented and mostly idle while users queue.
vLLM introduced two ideas that are now standard:
Continuous batching — new requests join the running batch as slots free up, instead of waiting for the whole batch to finish. No request waits for an unrelated one to complete.
PagedAttention — attention memory is managed in pages, like virtual memory in an operating system, instead of one contiguous block reserved per request. Far less waste, far more concurrent sequences on the same card.
The practical result is throughput several times higher on identical hardware.
Getting it running
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192It exposes an OpenAI-compatible endpoint, which is the detail that makes adoption cheap — existing client code points at a new base URL and keeps working.
When it is the right call
Use it when multiple users hit the same model concurrently, when you are serving a model as a shared internal service, or when GPU cost per request is a line item someone is watching.
Skip it for a single-user desktop setup, where the operational complexity buys you nothing over a simpler runner.
The catch
It expects a proper NVIDIA GPU and rewards VRAM. It is not the tool for a laptop.
Tune --max-model-len deliberately: it reserves memory proportional to the context you promise. Advertising 128k context when your prompts are 4k burns capacity you could have spent on concurrency.
And measure the metric that matters to a user — time to first token — not just tokens per second in aggregate. High throughput with slow first tokens feels broken even when the dashboard looks excellent.
Resources
- Repo: vllm-project/vllm
- Docs: docs.vllm.ai
- Video walkthroughs: YouTube: vLLM serving
Need this built properly?
I build secure, fast, bilingual platforms for clients across Egypt, Saudi Arabia, the UAE and Kuwait.
