One Gateway in Front of Every Model

The first LLM feature calls one provider's SDK directly. By the third feature you have that SDK in three services, the API key in three environment files, no idea what any of it costs, and a migration project if you ever want to change provider.

A gateway fixes the shape before it sets.

What it is

A proxy that exposes one OpenAI-compatible API to your applications and translates to whichever provider actually serves the request.

Copy codemodel_list:
  - model_name: fast
    litellm_params:
      model: anthropic/claude-haiku-4-5-20251001
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: smart
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

Your code asks for fast or smart. It never names a provider. Changing what fast means is a config edit and a restart, not a deployment across every service that calls a model.

The reasons that actually justify it

Keys stop spreading. Provider credentials live in the gateway. Applications get a gateway key you can rotate or revoke without touching the provider account. When a developer leaves, that is one revocation.

Budgets become enforceable.

Copy codecurl -X POST $GATEWAY/key/generate \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d '{"models": ["fast"], "max_budget": 50, "duration": "30d"}'

A per-team, per-client, per-environment key with a hard ceiling. A runaway loop in a staging job stops at the limit instead of appearing on the invoice.

You get one log. Every call, its cost, latency and error rate, in one place. The first time you look at that table you will find a feature costing far more than you assumed. That has been true every time I have set one up.

Fallbacks are configuration. Provider returns 429, the request routes to the alternate. Your application code has no retry logic in it.

The trade

It is a hop in front of every model call, so it is now on the critical path. Run it with the availability you would give any other dependency, and keep a documented direct path for the case where the gateway itself is down.

When to add it

Before the second service starts calling a model. Retrofitting a gateway after five services each grew their own client code is a week; putting it in early is an afternoon.

Resources

• Repo: BerriAI/litellm
• Docs: docs.litellm.ai
• Video walkthroughs: YouTube: litellm proxy setup tutorial
• Related: An LLM proxy that strips PII