LLM gateways and model routing: cutting cost without losing quality

Open the usage export of most AI products and you find the same pattern: one large model handles everything. It writes the complicated answers, and it also classifies tickets, extracts dates and decides whether a message is spam. As of October 2026 the small tier from both Anthropic and OpenAI lists at one twentieth of the large tier’s price.¹² Paying the large rate for routine work is the biggest avoidable line on most AI bills we review.

The fix has two parts. An LLM gateway puts one controlled layer between your code and every model provider. Routing uses that layer to send each request to the cheapest model that still passes your quality bar. This article covers both, with a worked cost example, a gateway config, a cascade sketch and the ways it goes wrong. It is platform-neutral; if you run on Bedrock, our Bedrock cost guide covers the platform-specific levers.

Prices and product details are as of October 2026. Model lineups and gateway features change monthly. Check each vendor’s pricing page before you budget.

What an LLM gateway does

A gateway is a reverse proxy that speaks model APIs. Your services call one endpoint, usually OpenAI-compatible, with a model alias such as small or triage. The gateway decides which provider and model serve it, and records what happened.

The side benefit is often the biggest: once every call goes through one layer, you finally know what you spend, on which feature, for which customer. Routing decisions without that data are guesses.

The options in October 2026

The market splits into self-hosted proxies, managed gateways, hosted aggregators and routers built into a cloud platform. These are the ones we meet most, with only facts we verified in each vendor’s documentation.

How we choose

• Data path. An aggregator or managed gateway is another processor of your prompts. If your data agreements name specific providers, a self-hosted proxy keeps the path short.
• Router transparency. Managed routers decide with a model you cannot inspect. The Foundry router’s effective context window is that of the smallest model in the pool, and a large prompt only succeeds if it happens to be routed to a model that can take it.¹³ Bedrock’s router cannot adjust its decisions based on your application’s own performance data.¹⁴ Rules you write yourself are less clever but fully testable.
• Supply chain. A gateway holds every provider key and sees every prompt, which makes it a prime target. In March 2026, two LiteLLM releases on PyPI (1.82.7 and 1.82.8) carried a credential stealer for part of a day; the maintainers advised anyone who installed them to rotate every secret on affected systems, and noted that the official Docker image, which pins dependencies, was not affected.¹⁵ Whatever you run, pin versions, verify artefacts and keep provider keys scoped.

Routing strategies

There are four strategies worth knowing. Most production systems we build combine two or three of them.

The research behind the last two is worth reading for the framing, not the headline numbers. FrugalGPT described the LLM cascade and reported matching its strongest model at a fraction of the cost on its benchmarks.¹⁶ RouteLLM trained routers on human preference data to choose between a strong and a weak model, and reported cost reductions of more than two times in some cases without losing quality.¹⁷ Both results are on their authors’ datasets. Yours will differ, which is why every routing change below goes through evals.

A gateway configuration

Here is a LiteLLM proxy config for a two-tier setup with a cross-provider fallback per tier. Keys follow the LiteLLM routing and reliability docs as of October 2026: model groups in model_list, fallbacks under router_settings, and timeouts and cooldowns under litellm_settings.³⁴⁶

model_list:
  # Aliases are what application code asks for. One model per alias,
  # so "small" always means the same behaviour and the same cache.
  - model_name: small
    litellm_params:
      model: anthropic/claude-haiku-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 3000
  - model_name: small-fallback
    litellm_params:
      model: openai/gpt-6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: large
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 1000
  - model_name: large-fallback
    litellm_params:
      model: openai/gpt-6-sol
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2                    # the only retry layer: set SDK retries to 0
  fallbacks:
    - {"small": ["small-fallback"]}
    - {"large": ["large-fallback"]}   # same tier, same list price; never a pricier tier

litellm_settings:
  request_timeout: 60
  allowed_fails: 3
  cooldown_time: 30
  cache: true
  cache_params:
    type: redis                     # exact match only; see the caching section

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  # Budgets on keys, teams and customers need Postgres (DATABASE_URL).

Two choices in that file are deliberate. Each alias maps to exactly one model, rather than load-balancing two different models behind one name, because mixing models in a group makes quality vary request by request. And every fallback stays within its price tier. A fallback chain that ends at the most expensive model turns a provider outage into a bill.

A cascade in Python

A cascade only works if you can tell, cheaply and reliably, that the small model’s answer is good enough. Structured tasks are the sweet spot. This sketch triages support tickets through the gateway above and records why each request escalated.

import os, random
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError

gw = OpenAI(base_url="http://llm-gateway.internal:4000",
            api_key=os.environ["GATEWAY_KEY"], max_retries=0)   # gateway owns retries

class Triage(BaseModel):
    category: Literal["billing", "bug", "account", "legal", "other"]
    priority: Literal["low", "normal", "urgent"]
    needs_human: bool
    summary: str = Field(max_length=280)

AUDIT_RATE = 0.02          # share of accepted small-model answers re-run on "large"

def ask(alias: str, ticket: str) -> Triage:
    resp = gw.chat.completions.create(
        model=alias,
        messages=[{"role": "system", "content": TRIAGE_PROMPT},
                  {"role": "user", "content": ticket}],
        response_format={"type": "json_object"},
        max_tokens=600,                     # leave room for any reasoning tokens
    )
    return Triage.model_validate_json(resp.choices[0].message.content)

def escalation_reason(t: Triage, ticket: str) -> str | None:
    """Deterministic checks only. Self-reported confidence is too poorly calibrated."""
    if t.priority == "urgent" and not t.needs_human:
        return "urgent_without_human"
    if mentions_legal_terms(ticket) and t.category != "legal":
        return "legal_keyword_mismatch"
    if account_is_enterprise(ticket) and t.priority == "low":
        return "enterprise_low_priority"
    return None

def triage(ticket: str) -> tuple[Triage, dict]:
    try:
        first = ask("small", ticket)
        reason = escalation_reason(first, ticket)
    except ValidationError:
        first, reason = None, "invalid_output"

    if reason is None:
        if random.random() < AUDIT_RATE:
            enqueue_shadow_check(ticket, first)          # async: compare with "large"
        return first, {"route": "small"}

    final = ask("large", ticket)
    return final, {"route": "large", "escalation_reason": reason}

The audit sample is what keeps a cascade honest. Re-running a small share of accepted answers on the large model, off the request path, gives you an ongoing agreement rate between tiers. When it drops, a model or prompt change has moved the boundary and your checks have not noticed.

Worked example: one million requests a month

List prices per million tokens as of October 2026, for the tiers used in the config above plus each provider’s frontier model:¹²

OpenAI’s figures are standard-tier rates for prompts up to 272K tokens; Haiku 5.5 moves to higher rates above 100K tokens.¹² Our assumptions, which are illustrative rather than measured: 1,000,000 requests a month, 3,000 input and 400 output tokens per request, no prompt caching (it lowers every scenario roughly in proportion), and gateway hosting costs excluded. Output counts include reasoning tokens; Haiku 5.5 has adaptive thinking on by default, so measure output length on your own traffic.¹⁸

The cascade’s cost is roughly $500 plus $10,000 times the escalation rate (plus the audit). That gives you its break-even points directly. It beats the all-large baseline unless more than 95% of requests escalate, but it only beats the simpler static split while escalation stays below about 47%. If your checks escalate half the traffic, static routing is cheaper and has lower latency.

Fallback pricing matters just as much. GPT-6 Astra lists at five times the large tier.² If a fallback chain sent just 5% of the baseline’s large-tier traffic to it during a bad month, that slice would cost $2,500 instead of $500, adding $2,000 to the bill without anyone deciding to spend it.

Proving quality held

A routing change is a model change for some share of your traffic. Treat it with the same discipline. Our sequence:

1. Offline evals per request class. Run the candidate route against a labelled set drawn from production, broken down by class. An aggregate pass rate hides the minority class that the small model fails. Our guide to retrofitting evals covers building that set from traces.
2. Shadow traffic. Send a copy of live requests to the new route, off the request path, and compare against what users actually received, with a calibrated judge or deterministic checks. Nobody sees the shadow output.
3. Canary. Move a small, fixed percentage of real traffic, watching the online signals below. Cloudflare’s percentage nodes and Portkey’s canary testing exist for exactly this.⁹⁷
4. Keep watching after rollout. Escalation rate, audit agreement, eval score by route, user feedback and re-ask rate, all broken down by route and model.

This only works if every request records which route and model actually served it. A gateway that rewrites the model silently is a gateway you cannot evaluate. We put the route, the escalation reason and the response model on the trace, as described in our guide to tracing agents with OpenTelemetry.

Caching: provider prompt caching vs semantic caching

Two very different things get called caching, and routing affects both.

Provider prompt caching

Providers cache the processed prefix of your prompt and bill reuse at a fraction of the input price. It is exact: Anthropic requires 100% identical prompt segments up to the cache breakpoint, and OpenAI routes requests to cache-holding machines using a hash of the initial tokens plus an optional prompt_cache_key.¹⁹²⁰ This is safe, because the model still runs on your actual input. The catch is that routing breaks cache locality. Microsoft’s router documentation says it plainly: caching only helps when the same model handles consecutive requests with overlapping prefixes.¹³ Load-balancing one conversation across models, providers or many API keys turns cache reads into cache writes. Keep a conversation on one model and deployment, and only move it on a deliberate escalation.

Gateway response caching

A gateway can return a stored response without calling a model at all. Exact-match caching, where the whole request must be identical, is predictable. Cloudflare’s cache works this way, with per-request headers to set the TTL, skip the cache or supply a custom cache key.²¹ It suits repeated, non-personalised requests such as classifying the same product description twice. Make sure the cache key includes the tenant, or one customer can receive another’s answer.

Semantic caching returns a stored answer for a request that is merely similar. That is where the risk lives. “Cancel my order” and “don’t cancel my order” embed close together. LiteLLM’s own documentation warns that its semantic cache is designed for single-shot prompts, that on multi-turn or agent traffic it can replay stale responses, and recommends exact-match caching for agents.²² Add staleness (the answer was right last week) and poisoning (someone deliberately seeds the cache with a crafted answer to a common question), and we only use semantic caching for read-only, non-personalised, low-stakes content, with short TTLs and per-tenant scoping.

Budgets and per-tenant limits

Routing lowers the average cost per request. Budgets cap the worst case, which is what finance actually asks about. Three layers work well together:

• Per-key rate limits on every service and environment, so a runaway loop in staging cannot exhaust production quota.
• Per-tenant spend budgets with a period, matched to what each plan includes. LiteLLM sets max_budget and budget_duration on keys and teams and attaches budgets to customers; requests fail once the budget is crossed.⁵
• Degrade before you refuse. For many products the right response to an exhausted budget is a cheaper model, not an error. Cloudflare’s budget-limit node does this, switching to a fallback branch when the quota is exceeded.⁹

Decide in advance what happens when the budget store is unreachable. Failing open keeps the product up and risks the bill; failing closed protects the bill and takes the product down. Either can be right, but it should be a decision, written down.

Failure modes

• Router misclassification. The router sends hard requests to the small model and nothing flags it. Watch eval scores and feedback per route, and keep an audit sample even when you route with a classifier.
• Fallbacks that behave differently. A fallback model is a different model. Haiku 5.5 rejects assistant prefill with a 400 error and wants temperature, top_p and top_k removed from requests; prompts written for an older model can fail outright.¹⁸ Sonnet 5.5 returns a 400 for forced tool choice.²³ Switching models mid-conversation can silently drop reasoning: Haiku 5.5 does not read thinking blocks from Sonnet 5.5 or Opus 5.5, and the API drops them without an error.¹⁸ Run your eval suite against every fallback, not only the primary.
• Different tokenizers and context windows. The same prompt is a different number of tokens on each provider, so budgets and max_tokens tuned on one model can truncate on another. Haiku 5.5 alone produces about 30% more tokens than Haiku 4.5 for the same text.¹⁸
• Retry storms. Gateway, SDK and framework each retry, and a provider brownout becomes a self-inflicted outage. Retry in one layer only.
• Cache poisoning and staleness. Covered above; mostly a semantic-cache problem, but exact caches without tenant in the key leak too.
• The gateway as a single point of failure. Run it redundantly, give it health checks and timeouts, and know how to bypass it in an emergency.
• Invisible routing. Logs record the alias requested, not the model that answered, so cost and quality analysis attributes results to the wrong model.

Checklist

1. All model traffic goes through one gateway; provider keys live only there.
2. Application code asks for aliases; each alias maps to exactly one model.
3. Every request logs alias, response model, route, escalation reason, tokens and cost.
4. Static routing for known simple step types, before anything cleverer.
5. Cascades only where a cheap deterministic check exists, with a shadow audit sample.
6. Break-even escalation rate computed from your prices and traffic.
7. Fallbacks stay within a price tier and are covered by the eval suite.
8. Retries happen in exactly one layer.
9. Conversations stay on one model and deployment to keep provider prompt caches warm.
10. Response caches are exact-match and tenant-scoped; semantic caching only for low-stakes, non-personalised content.
11. Per-key rate limits, per-tenant budgets and a written fail-open or fail-closed decision.
12. Routing changes go through offline evals, shadow traffic and a canary.
13. Gateway versions pinned and artefacts verified.

Routing is where we usually find the largest saving when we review an AI system, and also where quality quietly slips if nobody is measuring. If you want an independent look at yours, that is part of our AI System Audit.

Sources

1. Anthropic, Claude API documentation, “Pricing” — platform.claude.com/docs/en/about-claude/pricing
2. OpenAI API documentation, “Pricing” — developers.openai.com/api/docs/pricing
3. LiteLLM documentation, “Router — load balancing” (model groups, routing strategies, retries, cooldowns) — docs.litellm.ai/docs/routing
4. LiteLLM documentation, “Fallbacks, retries, timeouts, cooldowns” (proxy reliability) — docs.litellm.ai/docs/proxy/reliability
5. LiteLLM documentation, “Budgets, rate limits” (keys, teams, customers) — docs.litellm.ai/docs/proxy/users
6. LiteLLM documentation, “Caching” (proxy) — docs.litellm.ai/docs/proxy/caching
7. Portkey documentation, “AI Gateway” — portkey.ai/docs/product/ai-gateway
8. Cloudflare AI Gateway documentation, overview — developers.cloudflare.com/ai-gateway/
9. Cloudflare AI Gateway documentation, “Dynamic routing” (last updated 2 Oct 2026) — developers.cloudflare.com/ai-gateway/features/dynamic-routing/
10. OpenRouter documentation, “Model fallbacks” — openrouter.ai/docs/guides/routing/model-fallbacks
11. OpenRouter documentation, “Auto router” — openrouter.ai/docs/guides/routing/auto-model-selection
12. OpenRouter documentation, FAQ (pricing and fees) — openrouter.ai/docs/faq
13. Microsoft Learn, “Model router for Microsoft Foundry concepts” (updated 19 Oct 2026) — learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-router
14. AWS, “Understanding intelligent prompt routing in Amazon Bedrock” — docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html
15. LiteLLM, “Security update: suspected supply chain incident,” March 2026 — docs.litellm.ai/blog/security-update-march-2026
16. Chen, Zaharia and Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance,” arXiv:2305.05176, May 2023 — arxiv.org/abs/2305.05176
17. Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data,” arXiv:2406.18665, Jun 2024 — arxiv.org/abs/2406.18665
18. Anthropic, Claude API documentation, “Claude Haiku 5.5 migration guide” — platform.claude.com/docs/en/models/haiku-5-5/migration-guide
19. Anthropic, Claude API documentation, “Prompt caching” — platform.claude.com/docs/en/build-with-claude/prompt-caching
20. OpenAI API documentation, “Prompt caching” — developers.openai.com/api/docs/guides/prompt-caching
21. Cloudflare AI Gateway documentation, “Caching” (last updated 30 Sep 2026) — developers.cloudflare.com/ai-gateway/features/caching/
22. LiteLLM documentation, “Semantic caching” — docs.litellm.ai/docs/proxy/caching_semantic
23. Anthropic, Claude Platform release notes (Sonnet 5.5, 28 Sep 2026; Haiku 5.5, 7 Oct 2026) — platform.claude.com/docs/en/release-notes/overview

Prices, product features and configuration keys were checked against each primary source in October 2026. The worked example is our own arithmetic on list prices and stated assumptions, not measurements from a client system. Code and configuration are illustrative sketches rather than drop-in implementations.