LLM Gateway Architecture: Routing, Caching, Fallbacks and Budgets

Nobody sets out to build an LLM gateway. You ship one feature: one key, one SDK call inside a try/except. Then the fifth feature ships, the keys live in four places, and one provider’s bad hour is every feature’s bad hour.

Updated 15 October 2026 · Tested with Python 3.12, FastAPI 0.142, httpx 0.28, Redis 8.10.2

An LLM gateway (also called an AI gateway) is a reverse proxy between your services and the model providers that owns the keys, routing, retries, budgets and usage records for every call. LLM gateway architecture is the design of that one request path.

Why bother? On 11 December 2024, OpenAI’s API was degraded or down for 4 hours and 22 minutes (OpenAI’s write-up). A feature that called only that API had nowhere else to go.

The proxy is the easy part. My position: LLM gateway system design is failure handling on one request path. Who retries? When do you fail over? Who pays, and when do you say no? The patterns are the ones from the System Design series: timeouts, circuit breakers, idempotency keys, rate limits.

This is about the pattern, not the product called LLM Gateway. If you run LLM features in production, you’ll come away knowing whether you need a gateway yet, how to keep a provider outage from taking your features down, and how to stop retries and runaway spend from multiplying. Each pattern comes with Python you can run against a mock provider (the code for this post) and the numbers its tests measured, and the last section helps you choose between building and buying one.

What is an LLM gateway, and how does it work?

An LLM gateway is one service that every model call passes through. It authenticates the caller’s virtual key, applies policy, runs input guardrails, checks the cache, picks a route, reserves budget, calls the provider (with timeouts, retries, a circuit breaker and fallbacks), streams the answer back, then settles the real cost and records the call.

That list is the data plane, what happens on every request. The control plane behind it changes rarely: virtual keys, routes, a versioned price table, budget limits. Fast-moving state lives in Redis, so every replica sees the same numbers.

A virtual key is a key the gateway issues: it maps a caller to a team, a tenant and a feature, and lists the routes it may use. The provider’s real key never leaves the gateway.

Before: the refunds service holds a production key, retries every error five times and logs full prompts.
After: it holds a scoped virtual key, and the gateway owns retries, redaction and the answer to “what did refund summaries cost last month?”

The LLM gateway architecture, step by step

1. Authenticate. The virtual key names the team, tenant and feature. Unknown key: 401.
2. Apply policy. May this key call this route? If not, 403. The route also caps max_tokens.
3. Run input guardrails: redaction, abuse checks, size limits.
4. Check the cache, where the route allows it. A hit calls nobody.
5. Pick the route. A task name becomes an ordered chain of models.
6. Reserve budget: the priced worst case, before any tokens are bought. None left: 429.
7. Call the provider with your own timeouts, retries, a circuit breaker and a fallback. Every target down: 503.
8. Stream back. After the first token, a failure is reported inside the stream.
9. Settle and record: refund what wasn’t used, then write tokens, cost, latency and a trace.

Steps 6 and 7 repeat per target, because the worst case is priced per model. The gateway in the code folder does every step except 3. Asked to design an LLM gateway in a system design interview? Talk in this order: the request path, the state stores, then the failure modes.

LLM gateway vs API gateway, MCP gateway and agent gateway

An API gateway meters requests and routes them to your own services. An LLM gateway meters tokens and money and routes to model providers. An MCP gateway fronts tool servers, an agent gateway fronts other agents. Products blur the lines (“AI gateway” often covers the last three), so look at the unit being metered.

Do you even need an LLM gateway yet?

Probably not with one service, one provider and one team. A shared client library with sane timeouts, one retry policy and structured logs goes a long way, and provider controls (OpenAI’s project-level limits, Anthropic’s workspace limits) cover basic separation.

You probably do if two or more of these are true:

• Several teams or services call LLM providers.
• You use (or plan to use) several providers.
• Nobody can answer “what does feature X cost?”
• You need per-tenant limits, data rules or PII controls.
• An outage or rate limit has already taken a feature down.
• Provider keys live in more places than you can list.

Routing: how does the gateway pick a model?

By looking it up. The caller names a task, like support-reply, and the route table maps it to an ordered chain of provider and model, each with its own timeout and max_tokens cap (part of routes.py):

@dataclass(frozen=True)
class Target:
    provider: str         # which upstream: a key of config.PROVIDERS
    model: str            # that provider's model ID
    timeout_s: float      # one attempt, start to finish
    first_token_s: float  # streams only: how long to wait for the first token
    max_tokens: int       # the most output one call may buy here

    @property
    def name(self) -> str:
        return f"{self.provider}/{self.model}"


@dataclass(frozen=True)
class Route:
    chain: tuple[Target, ...]  # tried in order: the primary, then its fallbacks
    cache_ttl_s: int = 0       # 0 = never cache this route's answers


ROUTES: dict[str, Route] = {
    "support-reply": Route((
        Target("alpha", "alpha-large", timeout_s=30, first_token_s=3, max_tokens=1024),
        Target("beta", "beta-large", timeout_s=30, first_token_s=3, max_tokens=1024),
    )),
    "classify": Route((
        Target("beta", "beta-small", timeout_s=5, first_token_s=2, max_tokens=64),
        Target("alpha", "alpha-small", timeout_s=5, first_token_s=2, max_tokens=64),
    ), cache_ttl_s=300),
    "summarize": Route((
        Target("alpha", "alpha-large", timeout_s=60, first_token_s=5, max_tokens=2048),
    )),
}

Callers never see a model ID, so you can remap a task without touching a client. The usual strategies sit on top: by task, by cost, by latency, a canary slice for a new model, and fallbacks.

One warning on fallbacks. A fallback model is a different model, and your prompts, parsers and tool schemas were tuned on the primary. If you’ve never run your evals against it, you don’t have a fallback. You have a second outage waiting.

How is that different from an LLM router? A router picks the model for one prompt (RouteLLM sends simpler queries to cheaper models); a gateway enforces policy around every call and can hold a router as step 5. Routing also reads live state before it commits: the circuit breaker and the remaining budget. And routes aren’t only for chat: the embedding calls of a production RAG architecture can be routes too.

When a provider fails: retries, fallbacks and a circuit breaker

Timeouts, and which errors deserve a retry

Set your own timeouts, give the whole request one deadline, and decide per error. A 429 with Retry-After and a 5xx deserve a retry on the same provider. A spend-limit error, a rejected key and a timeout go to the next target. Any other 4xx is the caller’s bug: hand it back.

The official OpenAI and Anthropic Python SDKs both default to a 10-minute timeout and two retries, far too patient for a user watching a spinner. In this gateway each target has its own timeouts, and one deadline_s covers every attempt, backoff and fallback.

The Anthropic rows come from its rate limits page, the OpenAI ones from its rate limit guide and error code list. Every row has a test.

Why does one click turn into 27 upstream calls?

Because every layer retries, and retries multiply. Your code tries three times. The SDK underneath makes three attempts for each (its default is two retries). The gateway makes three upstream attempts per request. That’s 27 calls to a provider that is already down.

Let’s test it, with the circuit breaker switched off, so only the retry layers are counted. The layers are real: an application loop, the openai SDK with its defaults and the gateway, against a mock that always answers 503. In 20 runs the mock counted 27 calls every time, and the click took a median 11.4 seconds to fail. With max_retries=0 on the SDK: 9 calls. With the application’s loop gone too: 3 calls, 1.0 seconds.

An untouched default SDK also stops at 9 when the gateway answers with x-should-retry: false, a non-standard header both SDKs obey (it’s in their source).

So should the app or the gateway retry? The gateway. Clients run with max_retries=0, and application code retries only a gateway 503 that carries Retry-After, and then with an idempotency key. Now count the retry layers in your own stack.

The circuit breaker: stop waiting on a provider that’s down

A circuit breaker remembers that a provider is failing, so the next request doesn’t have to find out again. Closed: calls go through and failures are counted. Open: skip the provider without calling it. Half-open: when the open time is up, one request goes through as a probe, and its result closes or reopens the breaker.

Two details. First, how long to stay open. After three plain failures it is 10 seconds by default. On a Retry-After longer than 2 seconds this gateway opens the breaker for exactly that long, as Azure API Management’s breaker does (Microsoft Learn). A spend-limit or key error opens it at once, for 10 minutes.

Second, the probe. With ten replicas, half-open must not mean ten probes, so the state lives in Redis and the right to probe is a short-lived Redis lock (part of breaker.py):

    async def allow(self, name: str) -> Gate:
        opened, half, fails = await self.r.mget(
            self._key(name, "open"), self._key(name, "half"), self._key(name, "fails"))
        if opened:
            return Gate("open", 0)
        if not half:
            return Gate("closed", int(fails or 0))
        # Half-open: the open time has run out and nobody has seen a success yet. One
        # request across all replicas may try. SET NX is the lock, and its TTL frees it
        # if the prober dies. A lock lost that way costs one extra probe, nothing worse.
        won = await self.r.set(self._key(name, "probe"), "1", nx=True, px=self.probe_ms)
        return Gate("probe" if won else "open", 0)

If the prober dies, its lock expires after 5 seconds and one more request gets to try. Now the numbers: four copies of the gateway, one per defense, go through a 15-second outage of the primary, with one streamed request every 250 ms and a fallback that answers in 50 ms. An attempt may take 5 seconds, a first token 1 second, and the breaker opens for 5 seconds after three failures.

Read the hang columns first. A breaker alone still left 23 of 60 callers waiting 5 seconds, because it can only count a failure after it has happened; everyone whose call started before the breaker tripped was already waiting.

The first-token timeout alone makes every caller wait 1 second. Together, the first six or seven wait a second, and after that only the probes are slow.

For a 503 the breaker does all the work. Without it, every request still made three calls to the dead primary, with backoff sleeps in between: 180 calls in 15 seconds, against 5.

What happens when a stream fails halfway?

Before the first token, a fallback is invisible: the caller has seen nothing, so the gateway can switch providers and the answer just starts a little later. After the first token it can’t swap answers mid-sentence, so it sends an error event inside the stream and the client decides what to do with half an answer.

That takes two timeouts a plain HTTP client doesn’t have. A first-token timeout covers the wait for the first real token. An idle-gap timeout covers the silence between two chunks after that.

Anthropic’s error docs say an error can occur after the API has returned a 200. In this gateway, a mid-stream drop ends the stream with data: {"error": ...} and no [DONE], the OpenAI SDK raises on it, and the budget is settled on what was sent: three tokens in the test, not the eight reserved.

Per-tenant budgets: reserve first, settle after the call

A budget check that reads the counter, calls the model and adds the cost afterwards has a hole exactly one model call wide. Two requests see $5 left, each may cost up to $4, both pass, both spend $3, and the tenant is $1 over. The fix is to debit the worst case before the call, atomically, and refund the difference after it.

Reserve is one Lua script. Redis runs a script with nothing else in between, so no lock is needed. It checks every level (org, team, tenant, key) and debits all of them or none:

-- Reserve `amount` at every level of the hierarchy, or at none of them.
-- KEYS[1]  = the reservation
-- KEYS[2..] = one (spent, limit) pair per level: org, team, tenant, key
-- ARGV[1]  = amount in micro-USD: the priced worst case, input + max_tokens
-- ARGV[2]  = reservation TTL in seconds
if redis.call('EXISTS', KEYS[1]) == 1 then
  return {2, 0}                               -- this ID already holds a reservation: a retry
end
local amount = tonumber(ARGV[1])
for i = 2, #KEYS, 2 do
  local spent = tonumber(redis.call('GET', KEYS[i]) or '0')
  local limit = tonumber(redis.call('GET', KEYS[i + 1]) or '')
  if limit and spent + amount > limit then
    return {0, i / 2}                         -- level i/2 said no, and nothing was debited
  end
end
for i = 2, #KEYS, 2 do
  redis.call('INCRBY', KEYS[i], amount)
  redis.call('EXPIRE', KEYS[i], 3456000, 'NX')  -- a month's counter is kept for 40 days
end
redis.call('SET', KEYS[1], amount, 'EX', ARGV[2])
return {1, 0}

The amount is the priced worst case: every input token plus the request’s max_tokens (capped by the route), in micro-dollars, so Redis only adds integers. Settle swaps that hold for the real cost:

-- Swap a reservation for what the call really cost.
-- KEYS[1]  = the reservation
-- KEYS[2..] = the same spent counters, one per level
-- ARGV[1]  = actual cost in micro-USD
local reserved = redis.call('GET', KEYS[1])
if not reserved then
  return -1          -- settled already, or expired: the full amount stays charged
end
local refund = tonumber(reserved) - tonumber(ARGV[1])
for i = 2, #KEYS do
  redis.call('DECRBY', KEYS[i], refund)
end
redis.call('DEL', KEYS[1])
return refund

Three properties fall out of those 36 lines, each with a test. A second settle is a no-op, because the reservation key is gone. A reservation that is never settled (the gateway died mid-call) expires after 5 minutes and stays charged, which fails safe. And a retried request holds one reservation, not two. On Redis Cluster, a hash tag per org keeps the keys in one slot.

Let’s test it: 200 requests at once for a tenant whose budget covers 10, with a mock that takes 2 seconds per call, so all of them are in flight together.

In 20 runs, check-then-spend served a median of 200 of the 200 and spent 20 times the budget. The Lua version served exactly 10, refused 190 with a 429 and spent exactly the limit, every time. (In 6 of the bursts a few naive requests, 36 in all, got a 500 while waiting for a Redis connection.)

Budgets and rate limits are different tools. A budget is a ledger, money per month; tokens per minute is a rate, where token bucket vs sliding window is its own decision. And keep the billing record outside Redis: writing the spend to a durable ledger too is a dual write, fixed by a transactional outbox and an idempotent consumer.

Last point: say no before the provider does. A 429 with code budget_exceeded from your own gateway hits one tenant and says why; Anthropic’s spend-limit error hits all of them. A tenant that can’t afford the primary’s worst case is still served by a cheaper fallback.

Backpressure: what if the backup provider can’t take your traffic?

Then failing over makes things worse. Budgets protect money; backpressure protects capacity. Cap what is in flight per provider and per tenant, queue briefly, and reject early with the right code: 429 when one tenant is over its own limit, 503 when the gateway is saturated. The provider side is a semaphore with a short queue (part of admit.py):

    async def admit(self, provider: str) -> bool:
        """Take one of the provider's slots, waiting at most queue_wait_s for it.
        False means the provider is saturated: fail over or shed the request."""
        slots = self.slots[provider]
        if slots.locked() and self.waiting[provider] >= self.max_queue:
            return False  # the queue is full: reject now instead of joining it
        self.waiting[provider] += 1
        try:
            with anyio.fail_after(self.queue_wait_s):
                await slots.acquire()
            return True
        except TimeoutError:
            return False  # waited long enough
        finally:
            self.waiting[provider] -= 1

A request waits at most 200 ms for a slot. With the mock slowed to 5 seconds and a tenant limit of 2, five requests at once give two in flight and three 429s in under half a second. With a provider limit of 2, the overflow goes to the fallback, or gets a 503 with Retry-After: 1 when there is none. The counters are per process: with N replicas, give each 1/N.

Now the part that bites in a real failover: providers dislike sudden ramps. OpenAI can answer slow_down inside your limits, and its guide says that from 1 million input tokens per minute you should grow by no more than 50% every 15 minutes. By that rule, 1M to 3M is three steps, about 45 minutes (my arithmetic, not their number). Anthropic’s docs warn of acceleration limits too.

So keep the secondary warm with real traffic, shift in steps, and decide in advance what gives: batch jobs wait, or everyone gets a cheaper model.

Is it safe to retry an LLM call?

It depends on which retry. Gateway to provider is usually safe: the worst case is paying for two completions. Client to gateway spends the budget twice unless the gateway dedupes. Agent to tool is the dangerous one: replay a step whose tool already ran, and the refund goes out twice.

The gateway handles the middle one with an Idempotency-Key header, the same contract as idempotency keys on a payments API (part of idempotency.py):

async def claim(r: redis.Redis, tenant: str, key: str, body: dict) -> Claim:
    """Claim the key for this request, or raise Replay or Conflict."""
    fingerprint = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
    rkey = f"idem:{tenant}:{key}"
    mine = json.dumps({"state": "in_flight", "fp": fingerprint})
    # SET NX is the claim; GET returns what was already there, in the same round trip.
    existing = await r.set(rkey, mine, nx=True, ex=LEASE_S, get=True)
    if existing is None:
        return Claim(r, rkey, fingerprint)
    record = json.loads(existing)
    if record["fp"] != fingerprint:
        raise Conflict(422, "idempotency_key_reused",
                       "this Idempotency-Key was used with a different request body")
    if record["state"] == "in_flight":
        raise Conflict(409, "idempotency_key_in_flight",
                       "a request with this Idempotency-Key is still running")
    if record["state"] == "streamed":
        raise Conflict(409, "stream_not_replayable",
                       "this Idempotency-Key ran as a stream, and streams aren't stored")
    raise Replay(record["status"], record["body"])

SET NX with GET claims the key or returns what was there. A repeat with a different body is a 422, a repeat while the first is still running a 409, and a repeat after it finished gets the stored answer with no model call. Streams aren’t stored, so a repeated stream gets a 409 too. It plugs into FastAPI as a dependency, which also releases the key when the request fails (part of gateway.py):

async def idempotent(request: Request, body: dict = Depends(parsed_body),
                     caller: Caller = Depends(authenticate),
                     route: Route = Depends(allowed_route),
                     idempotency_key: str | None = Header(default=None)):
    """Claim the caller's Idempotency-Key, if it sent one. A repeat of a finished request
    gets the stored answer (Replay); a repeat that can't be served is a Conflict."""
    if idempotency_key is None:
        yield None
        return
    claimed = await claim(request.app.state.gw.redis, caller.tenant, idempotency_key, body)
    try:
        yield claimed
    except Exception:
        await claimed.release()  # the request failed: the same key may be tried again
        raise

The key doubles as the request ID, and with the tenant in front it is the budget reservation ID. In the test, the same key sent ten times at once makes one upstream call: one 200, nine 409s, one charge.

None of this protects a tool. If your code already ran issue_refund, replaying the step refunds twice, and one request fans out into many model calls in a ReAct agent loop. So retry model calls, never tool executions, and give side-effecting tools idempotency keys of their own.

How to build a minimal LLM gateway in Python

The gateway in this post is about 1,150 lines of Python and Lua: FastAPI and httpx behind one OpenAI-compatible endpoint, POST /v1/chat/completions, with Redis for shared state. A mock provider with failure switches stands in for the real ones, and 114 tests cover every path described above. The tests and the gateway expect Redis on localhost:56379; set LLMGW_REDIS_URL to use yours.

cd genai-infrastructure/01-llm-gateway
pytest -q
uvicorn mock_upstream:app --port 8001 &
uvicorn gateway:app --port 8000 &
curl -s localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer vk-demo-refunds-acme" \
  -d '{"model": "support-reply", "messages": [{"role": "user", "content": "hello"}]}'

The heart is the loop that tries one target (part of gateway.py):

async def try_target(gw: Gateway, target: Target, body: dict, max_tokens: int,
                     deadline: float, failures: list[str]) -> tuple[object, int]:
    """Call one target, retrying only what is worth retrying. Returns (result, calls
    made); the result is None when it is time to move down the chain."""
    s = gw.settings
    provider = s.providers[target.provider]
    for attempt in range(s.attempts):
        gate = await gw.breaker.allow(target.name)
        left = deadline - time.perf_counter()
        if gate.state == "open" or left <= 0:
            failures.append(f"{target.name}: " + ("circuit open" if left > 0 else "out of time"))
            return None, attempt
        try:
            if body.get("stream"):
                wait_s = target.first_token_s if s.first_token_timeout else target.timeout_s
                result = await open_stream(gw.http, provider, target, body, max_tokens,
                                           min(wait_s, left))
            else:
                result = await complete(gw.http, provider, target, body, max_tokens,
                                        min(target.timeout_s, left))
        except UpstreamError as err:
            verdict = err.verdict
            if verdict.action is Action.FAIL:  # the caller's own mistake: no retry, no fallback
                raise GatewayError(err.status, verdict.reason, "rejected by the provider",
                                   body=err.body) from None
            # Retry-After is a minimum. The jittered backoff goes on top of it.
            wait = (verdict.retry_after or 0) + backoff(s, attempt)
            too_long = (verdict.retry_after or 0) > s.max_wait_s
            open_for = STOP_OPEN_S if verdict.action is Action.STOP else None
            if too_long:  # the provider said when to come back: stay away until then
                open_for = verdict.retry_after
            await gw.breaker.failure(target.name, gate, open_for)
            retry = (verdict.action is Action.RETRY and not too_long and gate.state != "probe"
                     and attempt + 1 < s.attempts and time.perf_counter() + wait < deadline)
            if not retry:
                failures.append(f"{target.name}: {verdict.reason}")
                return None, attempt + 1
            await asyncio.sleep(wait)
        else:
            await gw.breaker.success(target.name, gate)
            return result, attempt + 1
    return None, s.attempts

Ask the breaker, work out what’s left of the deadline, make the call, and on failure decide. Only a RETRY with a short enough wait, attempts left and time left sleeps and loops; the rest return None and call_chain moves to the next target. The first-token timeout is the other piece worth seeing (part of upstream.py):

async def open_stream(client: httpx.AsyncClient, provider: Provider, target: Target, body: dict,
                      max_tokens: int, first_token_s: float) -> Stream:
    """Send the request and wait for the first token. Nothing has gone to the caller
    yet, so whatever fails in here can still fail over."""
    resp, opened = None, False
    try:
        with anyio.fail_after(first_token_s):
            request = _request(client, provider, target, body, max_tokens, True)
            resp = await client.send(request, stream=True)
            if resp.status_code != 200:
                await resp.aread()
                raise _error(resp)
            events, held = _events(resp), []
            async for line, payload in events:
                held.append((line, payload))  # the role chunk and pings wait here
                if has_token(payload):
                    opened = True
                    return Stream(resp, events, held)
            raise UpstreamError(CONNECTION)   # the stream ended without a single token
    except TimeoutError:
        raise UpstreamError(FIRST_TOKEN_TIMEOUT) from None
    except httpx.TransportError:
        raise UpstreamError(CONNECTION) from None
    finally:
        if resp is not None and not opened:
            await resp.aclose()

Everything up to the first real token sits inside one fail_after, and the chunks read so far are held back, so when the timeout fires nothing from this provider has reached the caller. Three build notes:

• Pin a patched Starlette (1.3.1 or later; 1.0.1 fixes the flaw under security below, 1.3.1 a later denial of service) and FastAPI, with hashes. A gateway is an HTTP server that holds keys.
• Emit usage as OpenTelemetry spans with the gen_ai.* names (telemetry.py).
• Know what it leaves out: an admin UI, guardrails, key rotation, a durable ledger. Those are the reasons to buy.

Does an LLM gateway add latency?

Yes, a little. One request at a time against a mock that answers instantly, this one adds 1.1 ms at the median as a bare proxy and 2.4 ms with every check on. Next to a model that takes a second to start talking, that’s noise.

Every number here comes from the tests in the code for this post, so yours will differ.

The first row is the gateway’s cost. The other two are a queue: the callers ask one Python process for more than the 230 to 375 requests a second it can do. The fix is more processes.

Two more things the averages hide. With a mock that takes a second to its first token (64 callers, about 64 requests a second), the median added latency was within a millisecond of zero, but the p99 was 420 ms higher. And when all 64 fire in the same instant, the median caller waited 103 ms longer through an empty connection pool and 375 ms through one holding 64 idle connections.

httpx with no gateway shows the same gap (about 80 ms against 370 ms per burst), so the likely cost sits in the client’s pool bookkeeping rather than in the gateway’s checks. That’s on this Windows machine with httpx 0.28.1 and httpcore 1.0.9, not measured on Linux. Test your own burst before you trust a median.

Vendor numbers disagree by orders of magnitude. Bifrost’s README claims 11 microseconds of added latency at 5,000 requests a second. LiteLLM’s own July 2026 benchmark, against a local mock, puts its beta Rust gateway at 0.7 ms added p99, Portkey at 2.3 ms, Bifrost at 4.5 ms and its own Python proxy at 257.7 ms. None of them, mine included, says what a real provider adds.

Caching, redaction and traces: what else belongs in the gateway?

Three things are easier in one place than in twelve services: an exact-match cache, the rule for what you send and what you store, and one usage record per call.

Exact-match and prompt caching

Hash the full request and return the stored response on an identical hit. The bug is almost always the key:

Before: key = hash(user_message)
After: key = hash(tenant, model, params, system_prompt, tools, messages)

Miss the system prompt and a prompt fix keeps serving old answers. Miss the tenant and one customer gets another customer’s answer. This gateway’s key covers the tenant, the model chain and the whole request, and every entry has a TTL: a response cache is user data at rest.

Provider prompt caching is different. OpenAI caches prompt prefixes automatically; Anthropic uses a cache_control field. Both cache the start of the prompt, not the answer, so keep system prompts and tools stable and first. On Anthropic, cache reads don’t count toward input-token rate limits for most models.

Semantic caching serves a stored answer to a merely similar prompt, and it’s dangerous exactly where it’s tempting: “cancel my subscription” and “don’t cancel my subscription” can sit close together in embedding space.

What you send vs what you store

Two decisions get merged that shouldn’t be: what you send to the provider and what you store. The model may need a customer’s name to draft an email; your logs almost never do.

Redact logs by default; redact what you send the provider per feature, often with reversible placeholders like [EMAIL_1]. Redaction isn’t free.

Uber’s GenAI Gateway redacts personal data on the way out and restores it in the response, and its write-up calls the redactor a challenge for both latency and answer quality. This gateway has no redaction step; its usage records leave prompts and completions out.

Usage, cost and traces

For every request, record at least:

• Who: team, feature, tenant, virtual key.
• What: model requested and served, route, cache hit, fallback, retries.
• How much: input, output and cached tokens, priced from a versioned price table (prices change).
• How it went: time to first token, total latency, status, provider request ID.

With streaming, OpenAI’s Chat Completions API only reports usage if you set include_usage in stream_options, in a final chunk that may never arrive if the stream is cut; Anthropic sends cumulative counts in message_delta events. So the gateway always asks for the usage chunk, and flags the record as an estimate when a stream ends early.

Attribute cost per feature, not just per team. Self-consistency multiplies tokens by design (see my prompt engineering techniques post), and multi-agent setups like AgentVerse multiply them again. Better to spot that on a dashboard than on an invoice.

For traces, use OpenTelemetry’s GenAI conventions (gen_ai.provider.name, gen_ai.usage.input_tokens and the rest). As of October 2026 they’re still marked Development, and since v1.42.0 (12 June 2026) they live in their own repository, which has no releases, so pin a commit, as telemetry.py does.

Build vs buy: which LLM gateway fits your constraints in 2026?

Choose by constraint, not by ranking. Must it run inside your network? On Kubernetes? Do you need budgets enforced, or only spend reported? Each answer removes half the list.

Three names come up in every search. LiteLLM Proxy is an LLM gateway you host yourself. OpenRouter is a hosted aggregator, a gateway you rent. Amazon Bedrock is a model platform, and AWS’s own gateway guidance puts LiteLLM in front of it.

Which gateways exist, as of October 2026

Two names are missing on purpose. Helicone has been in maintenance mode since Mintlify acquired it in March 2026, and TensorZero’s repository is archived.

Securing the gateway itself

The gateway holds every provider key you own, so treat it like a secrets manager that happens to proxy traffic. 2026 gave an example of each way that goes wrong, all from LiteLLM and all fixed since.

• Supply chain. On 24 March 2026, litellm 1.82.7 and 1.82.8 on PyPI carried a credential stealer, which LiteLLM believes came from a compromised Trivy dependency in its CI. LiteLLM says they were up for about 40 minutes, IONIX about three hours. The official Docker image pins its dependencies and wasn’t hit.
• A management endpoint. CVE-2026-42271 let any authenticated user, even with a low-privilege key, run arbitrary commands on the proxy host through its MCP test endpoints. Chained with Starlette’s CVE-2026-48710, the key isn’t needed either, as Horizon3.ai showed (Help Net Security). Fixed in LiteLLM 1.83.7 and Starlette 1.0.1; both are on CISA’s exploited list.
• Authentication. CVE-2026-42208 let an unauthenticated attacker reach the proxy’s database through SQL injection in its API-key check; fixed in 1.83.7, on the list since 8 May. CVE-2026-59822 let an unauthenticated attacker reach MCP tooling with a fabricated Authorization header. Fixed in 1.84.0; on the same list since 2 September.

So: pin packages with hashes and verify image signatures. Keep admin and test endpoints off the public network. Rotate the master key. Give the gateway an egress allow-list of provider hosts only: the stealer posted what it found to an outside domain, which that rule would block (my reading, not LiteLLM’s claim). And watch CISA’s list for your gateway and its web framework.

What you still own

Here’s the job on one page. Build or buy, every row is still yours.

After buying you still own the routing policy and its evals, budget approvals, data rules, reconciling costs against invoices, and the gateway itself: it needs redundancy, SLOs and someone on call.

Over to you: are you calling providers directly, running an off-the-shelf LLM gateway, or maintaining your own? What pushed you there: cost, rate limits, or a prompt in the logs? And tell me which failure I missed. For more posts like this, join the newsletter.