I got paged twice in the last two months because a single model dependency went down. One was a Copilot billing spike after a silent pricing change on a reasoning-heavy endpoint. The other was Anthropic’s Claude Fable 5 and Mythos 5, getting suspended on June 12, 2026, to comply with U.S. Department of Commerce export controls, only to be restored on July 1 once the controls were lifted. Nineteen days of an unavailable model, zero warning, and any app hardcoded to that one model string was dead in the water until Anthropic’s team flipped the switch back on (official statement).

That’s the whole argument for this post. If your app calls one model, one provider, one endpoint, you don’t have an LLM integration; you have a single point of failure with a chat interface bolted on. This is a code-first tutorial on building a router that treats models like a pool of interchangeable workers: try the cheapest one that can do the job, fail over cleanly when it errors out or times out, and logs what it costs you. No frameworks you have to trust blindly, no hand-waving. Just Python that you can read in one sitting.

The problem with “just call the API”

Most tutorials show you this:

import openai

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Summarize this contract."}]
)

That’s fine for a demo. It’s not fine for production, because it fails on four separate axes:

1. Provider outage — OpenAI or Anthropic has a bad five minutes, and your whole app goes down with it.
2. Rate limits — you get 429’d during a traffic spike and have no plan B.
3. Cost drift — you’re paying frontier-model prices for a task that a cheaper model handles fine.
4. Model deprecation or access changes — a model gets deprecated, paused for compliance reasons, or moved behind a waitlist, like what happened with Fable 5.

A fallback chain fixes all four at once. The idea: define an ordered list of models, cheapest-and-fastest first, and walk down the list on any failure until one succeeds.

Step 1: Define your model chain as data, not code

Don’t hardcode if openai_fails: try_anthropic(). Model the chain as a config so you can reorder, add, or pull a model without touching your call logic.

from dataclasses import dataclass
from enum import Enum

class Provider(Enum):
    OPENAI = "openai"
    ANTHROPIC = "anthropic"

@dataclass
class ModelSpec:
    provider: Provider
    model_id: str
    input_cost_per_mtok: float   # USD per million input tokens
    output_cost_per_mtok: float  # USD per million output tokens
    max_retries: int = 2
    timeout_s: float = 30.0

# Ordered cheapest-capable-first. Reorder this list, don't rewrite your router.
FALLBACK_CHAIN = [
    ModelSpec(Provider.ANTHROPIC, "claude-sonnet-5", 2.00, 10.00),   # intro pricing through Aug 31, 2026
    ModelSpec(Provider.OPENAI, "gpt-5.5", 5.00, 30.00),
    ModelSpec(Provider.ANTHROPIC, "claude-haiku-4-5-20251001", 1.00, 5.00),
]

A note on the pricing numbers: Claude Sonnet 5 launched at an introductory rate of $2/$10 per million input/output tokens through August 31, 2026, moving to $3/$15 after that. GPT-5.5 runs $5/$30 per million tokens. If you’re on the trusted-partner preview for OpenAI’s GPT-5.6 family (Sol, Terra, Luna), Sol prices at $5/$30, Terra at $2.50/$15, and Luna at $1/$6 — but as of this writing GPT-5.6 is still limited-preview only, not self-serve API access, so don’t build a production chain that depends on it being generally available yet. Build the chain so swapping gpt-5.5 for gpt-5.6-sol later is a one-line config change, not a rewrite.

Step 2: Write one adapter per provider, same interface out

The point of the router is that call sites never know which provider actually answered. Wrap each SDK behind an identical function signature.

import time
import openai
import anthropic

openai_client = openai.OpenAI()
anthropic_client = anthropic.Anthropic()

class ModelError(Exception):
    """Raised on any failure we want the router to catch and fail over on."""
    pass

def call_openai(spec: ModelSpec, messages: list[dict]) -> tuple[str, int, int]:
    try:
        resp = openai_client.chat.completions.create(
            model=spec.model_id,
            messages=messages,
            timeout=spec.timeout_s,
        )
        text = resp.choices[0].message.content
        return text, resp.usage.prompt_tokens, resp.usage.completion_tokens
    except (openai.APIStatusError, openai.APITimeoutError, openai.RateLimitError) as e:
        raise ModelError(f"OpenAI {spec.model_id} failed: {e}") from e

def call_anthropic(spec: ModelSpec, messages: list[dict]) -> tuple[str, int, int]:
    # Anthropic's SDK wants a separate system prompt and no "system" role in messages
    system_msg = next((m["content"] for m in messages if m["role"] == "system"), None)
    user_msgs = [m for m in messages if m["role"] != "system"]
    try:
        resp = anthropic_client.messages.create(
            model=spec.model_id,
            max_tokens=4096,
            system=system_msg,
            messages=user_msgs,
            timeout=spec.timeout_s,
        )
        text = resp.content[0].text
        return text, resp.usage.input_tokens, resp.usage.output_tokens
    except (anthropic.APIStatusError, anthropic.APITimeoutError, anthropic.RateLimitError) as e:
        raise ModelError(f"Anthropic {spec.model_id} failed: {e}") from e

DISPATCH = {
    Provider.OPENAI: call_openai,
    Provider.ANTHROPIC: call_anthropic,
}

Notice I’m only catching the exception types that mean “this provider had a problem,” not blanket except Exception. If your prompt itself is malformed, you want that to blow up loudly, not silently retry three providers with the same broken request.

Step 3: The router — walk the chain, retry with backoff, track cost

This is the core piece. It does three things every fallback router needs: retries within a model before giving up on it, moves to the next model on total failure, and returns cost data so you can actually see what happened.

import logging

logger = logging.getLogger("llm_router")

def estimate_cost(spec: ModelSpec, input_tokens: int, output_tokens: int) -> float:
    return (
        (input_tokens / 1_000_000) * spec.input_cost_per_mtok
        + (output_tokens / 1_000_000) * spec.output_cost_per_mtok
    )

def route_request(messages: list[dict], chain: list[ModelSpec] = FALLBACK_CHAIN) -> dict:
    errors = []

    for spec in chain:
        fn = DISPATCH[spec.provider]

        for attempt in range(1, spec.max_retries + 1):
            try:
                start = time.monotonic()
                text, in_tok, out_tok = fn(spec, messages)
                latency = time.monotonic() - start
                cost = estimate_cost(spec, in_tok, out_tok)

                logger.info(
                    "success model=%s attempt=%d latency=%.2fs cost=$%.5f",
                    spec.model_id, attempt, latency, cost,
                )
                return {
                    "text": text,
                    "model_used": spec.model_id,
                    "provider": spec.provider.value,
                    "attempt": attempt,
                    "latency_s": round(latency, 3),
                    "cost_usd": round(cost, 6),
                    "fallback_depth": chain.index(spec),
                }

            except ModelError as e:
                errors.append(str(e))
                logger.warning("attempt %d/%d failed for %s: %s",
                                attempt, spec.max_retries, spec.model_id, e)
                if attempt < spec.max_retries:
                    time.sleep(min(2 ** attempt, 8))  # exponential backoff, capped at 8s

    # Every model in the chain failed
    raise ModelError(f"All models in chain exhausted. Errors: {errors}")

Run it like this:

result = route_request([
    {"role": "system", "content": "You are a concise contract-review assistant."},
    {"role": "user", "content": "Flag any auto-renewal clauses in this text: ..."},
])

print(result["model_used"], result["cost_usd"], result["fallback_depth"])

If Sonnet 5 answers on the first try, fallback_depth is 0 and you paid Sonnet 5 rates. If Sonnet 5 is rate-limited and GPT-5.5 picks it up, fallback_depth is 1 and you know exactly why your bill for that request looks different. That visibility is the whole point — you can’t optimize cost you can’t see.

Step 4: Don’t retry a provider that’s clearly dead — add a circuit breaker

Retrying a model that’s mid-outage on every single request wastes time and money, and it can make an outage worse by hammering an already-struggling API. A circuit breaker tracks failure rate per model and skips it entirely for a cooldown period once it crosses a threshold.

from collections import deque
from datetime import datetime, timedelta

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 3, window_s: int = 60, cooldown_s: int = 120):
        self.failure_threshold = failure_threshold
        self.window = timedelta(seconds=window_s)
        self.cooldown = timedelta(seconds=cooldown_s)
        self.failures: dict[str, deque] = {}
        self.tripped_until: dict[str, datetime] = {}

    def is_open(self, model_id: str) -> bool:
        """True means 'circuit open' — skip this model."""
        until = self.tripped_until.get(model_id)
        if until and datetime.now() < until:
            return True
        return False

    def record_failure(self, model_id: str):
        now = datetime.now()
        q = self.failures.setdefault(model_id, deque())
        q.append(now)
        while q and now - q[0] > self.window:
            q.popleft()
        if len(q) >= self.failure_threshold:
            self.tripped_until[model_id] = now + self.cooldown
            logger.error("circuit OPEN for %s until %s", model_id, self.tripped_until[model_id])

    def record_success(self, model_id: str):
        self.failures.pop(model_id, None)
        self.tripped_until.pop(model_id, None)

breaker = CircuitBreaker()

Wire it into route_request by checking breaker.is_open(spec.model_id) before you call fn, and calling breaker.record_failure / breaker.record_success inside the try/except. Three failures inside 60 seconds and that model gets skipped for two minutes — long enough to ride out a transient blip, short enough that you’re back on the cheaper model the moment it recovers.

Step 5: Skip building your own SDK glue — know when to use LiteLLM

Everything above is maybe 120 lines, and I’d argue you should understand it even if you don’t ship it. But if you want the multi-provider adapter layer without owning it, LiteLLM is a real, actively maintained open-source library that gives you one OpenAI-compatible interface across 100+ providers, plus built-in retry and fallback config:

from litellm import completion

response = completion(
    model="claude-sonnet-5",
    messages=[{"role": "user", "content": "Summarize this contract."}],
    fallbacks=["gpt-5.5", "claude-haiku-4-5-20251001"],
    num_retries=2,
)

That single call does most of Step 2 and Step 3 for you. What LiteLLM doesn’t hand you out of the box is the cost-aware ordering logic and the circuit breaker tuned to your own traffic patterns — that’s still on you to configure well. Use it when you want to move fast and trust a well-tested library; write your own router when you need tight control over ordering logic, custom cost accounting, or you’re operating in an environment where adding a dependency needs a security review.

Step 6: Route by task, not just by failure

The chain above fails overreactively — only after something breaks. The bigger cost win is routing proactively: send simple tasks to cheap models by default, and only escalate to the expensive model when the task actually needs it.

import re

def pick_chain_for_task(prompt: str, base_chain: list[ModelSpec]) -> list[ModelSpec]:
    token_estimate = len(prompt.split())
    is_complex = bool(re.search(r"\b(analyze|architecture|refactor|multi-step|reason)\b", prompt, re.I))

    if token_estimate < 200 and not is_complex:
        # cheap-first order, no need to touch the frontier model at all
        return sorted(base_chain, key=lambda s: s.input_cost_per_mtok)
    return base_chain  # keep the default, capability-first order

This is the difference between a router that just survives outages and one that actually cuts your bill. A classification prompt, a short summarization job, a form-field extraction — none of that needs a $30/million-output-token model. Save the expensive model for the requests that genuinely need deep reasoning, long context, or agentic tool use.

Step 7: Log everything, then actually look at it

The router above already returns model_used, cost_usd, latency_s, and fallback_depth per call. Pipe that into whatever you already use for logs — even a flat file with one JSON line per request is enough to start:

import json

def log_request(result: dict, path: str = "llm_router.log"):
    with open(path, "a") as f:
        f.write(json.dumps({**result, "ts": datetime.now().isoformat()}) + "\n")

After a week of real traffic, run a quick pass over that log:

import pandas as pd

df = pd.read_json("llm_router.log", lines=True)
print(df.groupby("model_used")["cost_usd"].sum())
print(df.groupby("model_used")["fallback_depth"].mean())
print((df["fallback_depth"] > 0).mean())  # % of requests that needed a fallback at all

If that last number is above a few percent, either your primary model has a real reliability problem worth escalating to the provider, or your primary choice in the chain is wrong for your traffic mix. Either way, you now have data instead of a hunch.

Things that will bite you if you skip them

• Token counting differs by tokenizer. Sonnet 5 uses an updated tokenizer versus Sonnet 4.6 — the same prompt can produce noticeably more tokens on the new model. Don’t reuse an old token-count baseline to estimate cost on a new model; re-measure.
• Streaming breaks naive fallback. If you’re streaming tokens to a UI and the model fails mid-stream, you can’t just “retry the next model” cleanly — you’ll show a broken partial response. Buffer the first chunk before committing to a stream, or fail over before you start streaming at all.
• Don’t fail over on content you disagree with. A model returning an answer you don’t like isn’t a ModelError. Only fail over on actual transport/API failures — timeouts, rate limits, 5xx errors. Otherwise you’ll burn budget hunting for the answer you wanted instead of handling real outages.
• Rate limit errors need real backoff, not a fixed sleep. Respect Retry-After headers when providers send them instead of guessing your own backoff window.
• Test the failure path, not just the happy path. Point your primary model at a bad API key on purpose in staging and confirm the chain actually falls through. A fallback chain you’ve never seen fail is a fallback chain you don’t actually know works.

The Bottom Line

A fallback chain is not a nice-to-have resiliency pattern you get to later. It’s the difference between an outage or a compliance-driven pause taking your product down for nineteen days versus your users never noticing. Build it as data (the chain config), keep provider-specific code behind identical adapters, retry smart, circuit-break aggressively, route cheap-first when the task allows it, and log enough that “what did this cost us and why” is a query, not a guess.

The code above is a complete, working starting point, not pseudocode. Copy it, swap in your own model chain, and you’ve got a router that survives the next model shutdown, price hike, or export-control surprise without a rewrite.