Tutorials

LLM fallbacks: build the safety net before you need it

Sunny Sharma12 min read

In short

Model Router series · Part 5 of 6 · Start from Part 1 · All parts ↓

Part 4 ended with a route that is a single bet. This part adds a safety net, one layer at a time, and tests it by breaking things on purpose.

  • Every layer is a new price, so every layer needs a limit. A fallback chain tries the same tier, a second provider, then the next tier up. A cap of four attempts and an escalation budget stop a bad day from becoming the bill.
  • A cheap tier may not write. If gpt-5.4-nano tries to raise a ticket, the turn moves up before the ticket is written. Once a write may have run, the turn is never re-run on another model, whether the check failed or the call errored. If the outcome is unknown, the turn stops with an error.
  • The cascade never fired on the held-out questions. Zero escalations in 96 runs. Its check reads only the answer text. The question the router loses in Part 4 gives the right number without the handbook search, and no answer check can see that.

Where this fits: part 5 of 6. We replace the executor in the gateway. The next part logs and proves the result.

Paper cut-out transit map with dotted transfer lines added, showing an LLM fallback to another provider when a line fails

The model the router picks can be slow, rate limited or down, and a cheap model sometimes answers badly. Here is a safety net for both, starting with an LLM fallback to another provider, and then I break things on purpose to see if it holds. One limit up front: I did not run a live second provider. The fallback to another provider is tested with scripted models, and the code path is the same. This layer is insurance, and the held-out test set does not exercise it.

How a routed call fails

Five ways, and each needs a different answer:

  1. A timeout. The model does not answer in time.
  2. A rate limit. The provider says slow down.
  3. A provider error. A 500, or a bad request.
  4. An empty or broken answer. The model replies, but with nothing useful.
  5. A wrong route. The router picked a model that cannot do the job.

You already have defence for the first three at the call level. Agents Part 11 sets a 30 second timeout and two retries on every call, and the repo keeps those settings for any provider:

def resilient_factory(spec: str):
    """The Part 11 settings, for any provider: a 30 second call timeout and two retries inside the call."""
    return get_model(spec, timeout=30, max_retries=2)

From router/resilience.py.

What is new is what happens after those retries run out, and what to do about the last two failure types.

LLM fallback to another provider

If a model fails, the executor tries the next one in a short chain: the routed model, then the same tier at a second provider if you have one, then the next tier up.

Paper cut-out chain of three fallback steps: same tier, same tier at another provider, next tier up
    def chain(self, gateway: Gateway, tier: str) -> list[tuple[str, str]]:
        """Models to try for a tier, in order: the tier itself, the same tier at another provider,
        then the next tier up. Each item is (spec, tier)."""
        chain = [(gateway.catalog.tier_spec(tier, gateway.provider), tier)]
        if self.second_provider:
            other = "anthropic" if gateway.provider == "openai" else "openai"
            chain.append((gateway.catalog.tier_spec(tier, other), tier))
        if up(tier) != tier:
            chain.append((gateway.catalog.tier_spec(up(tier), gateway.provider), up(tier)))
        return chain

    def _attempt(self, gateway: Gateway, spec: str, tier: str, messages: list) -> _Attempt:
        guard = TIER_ORDER.index(tier) < TIER_ORDER.index(self.write_tier)
        ran: list = []
        tools = block_writes(TOOLS) if guard else record_writes(TOOLS, ran)
        with get_usage_metadata_callback() as usage:
            try:
                out = build_agent(gateway.model_factory(spec), tools=tools).invoke({"messages": messages})["messages"]
            except EscalateBeforeWrite:
                return _Attempt("blocked-write", usage=usage.usage_metadata)
            except Exception as exc:  # timeouts, rate limits, provider errors, bad requests
                err = f"{type(exc).__name__}: {exc}"[:200]
                if ran:   # a write may have run, so the outcome is unknown: do not try another model
                    return _Attempt("error-after-write", usage=usage.usage_metadata, error=err)
                return _Attempt("error", usage=usage.usage_metadata, error=err)
        return _Attempt("ok", out, usage.usage_metadata)

From router/resilience.py.

A hard limit on attempts, four by default, covers all of it. Without a limit, a cascade of fallbacks on a bad day can turn one request into a dozen calls.

The second provider is optional, and it comes with two cautions. Claude Sonnet 5.5 is priced close to gpt-5.4, not gpt-5.4-mini, so as a fallback behind mid it is a step up in cost. Test the fallback model on your own question set before you trust it. And a second provider is a second processor of your HR data. Check that your policies allow it. Two providers also means two output formats, which LangChain handles in one adapter.

The cascade

A cascade tries the cheap model first and escalates only if a cheap check says the answer is bad. The check looks only at the answer. It is not a second model call.

Paper cut-out flow of the cascade: try small, check the answer, go to mid on any yes
def check_answer(messages: list) -> list[str]:
    """Cheap checks on a finished turn. Returns the reasons it failed, or an empty list.

    1. The answer is not empty.
    2. The answer is not raw JSON or a tool dump.
    3. If the handbook search returned sections, the answer cites one in square brackets.
    4. The model did not refuse after a tool had already returned the data.
    The model's own "low confidence" is not used. It has not been shown to track correctness.
    """
    if not messages:
        return ["no messages"]
    last = messages[-1]
    text = last.content if isinstance(last.content, str) else str(last.content)
    failures = []
    if not text.strip():
        failures.append("empty answer")
    if text.lstrip().startswith(("{", "[{")):
        failures.append("answer is raw JSON")
    calls = [c["name"] for m in messages if m.type == "ai" for c in m.tool_calls]
    found_sections = any(m.type == "tool" and '"section"' in m.content for m in messages)
    if "search_policy" in calls and found_sections and not re.search(r"\[[^\]]+\]", text):
        failures.append("handbook searched but no section cited")
    got_data = any(m.type == "tool" and '"ok": false' not in m.content.lower() and '"results": []' not in m.content
                   for m in messages)
    if got_data and REFUSAL.search(text):
        failures.append("refused although a tool returned data")
    return failures

From router/resilience.py.

One check I left out on purpose: the model’s own “I’m not sure”. Nothing shows it tracks whether the answer is right.

What it cannot do: see a confident wrong number. A cascade only catches the failures its check can see.

A cheap tier must not write

This is the rule I would defend hardest. If a cheap model is about to raise a ticket, and a quality check then fails, escalating would run the whole turn again on a bigger model, and the bigger model might raise a second ticket. An idempotency key does not rescue you, because a second model may call the tool with different arguments.

Paper cut-out gate that blocks a cheap model's ticket write and hands over to mid

So the cheap tiers get a version of the tools where the write tool raises instead of running:

def block_writes(tools: list, write_names=WRITE_TOOLS) -> list:
    """The same tools, except write tools raise EscalateBeforeWrite instead of running."""
    def blocked_version(tool):
        def refuse(**kwargs):
            raise EscalateBeforeWrite(tool.name)
        return StructuredTool.from_function(func=refuse, name=tool.name, description=tool.description,
                                            args_schema=tool.args_schema)
    return [blocked_version(t) if t.name in write_names else t for t in tools]

From router/resilience.py.

When it fires, the turn moves up a tier before anything is written. And once a write may have run, the turn is never re-run on another model. A failed check does not re-run it. An error does not either. My first version of the executor got the error case wrong. If a mid model raised a ticket and the next call timed out, the chain treated it as a failed attempt and tried the next model, and when that model worded the ticket differently, two tickets existed. The write tools now note that they were called, so an exception after a write stops the turn with an error instead of a retry. The outcome of that write is unknown, and the right next step is to look it up, not to ask another model. A real backend should also take an idempotency key per request, and keep the write behind the approval and security controls from Agents Part 7 and Part 10. Scenario 6b in the script shows it.

A budget for escalations

On a bad day the cheap model might fail the check on most requests. Every one would escalate, and you would silently be paying for the large model on everything. An escalation budget caps that:

Paper cut-out grid of 100 requests with 25 marked as the escalation budget
class EscalationBudget:
    """At most max_rate of the last `window` requests may escalate for quality reasons.

    Escalating before a write is not counted: it is needed for correctness, not a choice.
    """

    def __init__(self, max_rate: float = 0.25, window: int = 100):
        self.max_escalations = max_rate * window
        self.recent: deque[bool] = deque(maxlen=window)

    def allow(self) -> bool:
        return sum(self.recent) < self.max_escalations

    def record(self, escalated: bool) -> None:
        self.recent.append(escalated)

    @property
    def used(self) -> int:
        return sum(self.recent)

From router/resilience.py.

The default is 25 percent of the last 100 requests. Past that, the executor serves the answer it has and notes why.

Here is the executor that ties it together:

    def __call__(self, gateway: Gateway, route: Route, messages: list) -> Outcome:
        usage: dict = {}
        attempts: list[dict] = []
        tier, escalated, quality = route.tier, False, False   # quality: an escalation the budget pays for
        while True:
            result = None
            for spec, spec_tier in self.chain(gateway, tier):
                if len(attempts) >= self.max_attempts:
                    break
                a = self._attempt(gateway, spec, spec_tier, messages)
                usage = merge_usage(usage, a.usage)
                attempts.append({"spec": spec, "tier": spec_tier, "ok": a.status == "ok", "status": a.status,
                                 "error": a.error})
                if a.status == "error-after-write":
                    attempts[-1]["note"] = "a write may have run, so no other model is tried"
                    break
                if a.status != "error":
                    result, tier = a, spec_tier
                    break
            if result is None:
                self.budget.record(quality)
                return Outcome([], usage, attempts, escalated,
                               error=(attempts[-1]["error"] or "attempt limit reached") if attempts else "no attempts allowed")
            if result.status == "blocked-write":
                tier, escalated = up(tier), True     # needed for correctness: not charged to the budget
                attempts[-1]["note"] = "cheap tier tried to write, moved up before the write ran"
                continue
            problems = self.check(result.messages) if self.cascade else []
            if not problems:
                self.budget.record(quality)
                return Outcome(result.messages, usage, attempts, escalated)
            attempts[-1]["check_failed"] = problems
            if wrote(result.messages):
                attempts[-1]["note"] = "a write already ran, so this turn is not re-run"
            elif tier == "large" or len(attempts) >= self.max_attempts:
                attempts[-1]["note"] = "no higher tier or attempt left"
            elif not self.budget.allow():
                attempts[-1]["note"] = "escalation budget used up, serving this answer"
            else:
                tier, escalated, quality = up(tier), True, True
                continue
            self.budget.record(quality)
            return Outcome(result.messages, usage, attempts, escalated)

From router/resilience.py.

One cost to know about. A check after the answer breaks the token-by-token streaming from Agents Part 11, because you cannot check an answer you have not finished receiving. If you stream to users, run the check on the finished answer and accept that an escalation shows up as a longer wait.

Break it on purpose

Hoping the safety net works is not testing it. The check script has eight scripted failures. A fake model fails on command, so you need no key.

python -m scripts.check_p5_failures
What goes wrongWhat happensEscalatedTickets written
The primary model times out, a second provider is setgpt-5.4-nano errors, claude-haiku-5-5 answersno0
The primary times out, no second providergpt-5.4-nano errors, gpt-5.4-mini answersno0
Every model failsBoth models fail, the turn stops and reports TimeoutErrorno0
The cheap model returns an empty answerCheck fails, mid answersyes0
The cheap model tries to raise a ticketBlocked before the write, mid answersyes1
A write already ran and the check failsNot re-runno1
A write ran, then the model errorsgpt-5.4-mini errors after the write, no other model is tried, the turn reports TimeoutErrorno1
The check fails on every request, budget of 2 per 10Requests 1 and 2 escalate, request 3 does not

The ticket count in the fifth row is the point. The cheap model was stopped before the write, mid wrote the single ticket, and the count is one, not two. The sixth row shows the other half: the turn already wrote, the check failed, and nothing re-ran. The seventh row is the error case: the model failed after the write, and the turn stopped with one ticket.

The last row shows the budget doing its job. Request 3 printed “escalation budget used up, serving this answer”.

These are scripted models, so they prove the wiring. The rejected answer in the fourth row is scripted too: no real model answer failed the check in any run. They say nothing about how a real model behaves. For that, the script also breaks the primary model for one real question. I made gpt-5.4-nano fail with a simulated timeout on “How many earned leave days do I have?”.

gpt-5.4-nano: error (TimeoutError)  ->  gpt-5.4-mini: ok
cost $0.001137   answer: You have **12 earned leave days** remaining.

The nano attempt failed before using any tokens, so the turn cost what a mid answer costs: $0.001137, the same as the mid price for that question in Part 1. The failure was free in dollars, and the fallback was the price of the tier above. A real timeout would also wait for the client timeout on each try before the fallback runs, and I did not measure that latency.

What the net costs, on the held-out questions

The last section of the check runs the held-out questions through the rules router with the cascade on.

SetupPassed$ per conversation$ per correct
Rules router (Part 3)95 of 96$0.00135$0.00137
Rules router + cascade95 of 96$0.00132$0.00134

Zero of 96 runs escalated. The cascade did nothing, because nothing failed its check.

Now the cost columns. The two rows send every question to the same tiers (24 small, 60 mid, 12 large) and the cascade never fired, yet the cost differs by about 2 percent. That gap is run-to-run variation, since model output and tool use differ a little each time. It is as large as the whole Part 3 saving, so read small differences with care anywhere in this series.

So did the safety net pay for itself? On this traffic it neither cost anything nor saved anything. Its value is insurance: the forced failure moved to a working model, and the write guard stopped a duplicate ticket. Insurance is hard to price from 96 calm runs.

The cascade’s blind spot is easy to name. In Part 4, the router lost three runs on one question, r-earned-8m, where nano gave the right number, “12”, without searching the handbook. A fluent answer with a plausible number passes every check above. The final setup in Part 6 puts this same cascade on top of the classifier, and it again passed 93 of 96 with no escalations. The cascade added nothing there either.

From a test to a log

The wiring holds up. How do we run this, watch it, and prove the number went down? That is Part 6.

Next: Part 6: Prove your LLM router saves money

The series

  1. Not every prompt needs your biggest model
  2. How do you know the small model is good enough?
  3. Your first LLM router is just a few rules
  4. When the rules run out, let a small model decide
  5. LLM fallbacks: build the safety net before you need it (this part)
  6. Prove your LLM router saves money
Sunny Sharma

Security and engineering leader. Notes from running AI, Azure and security in production.

Leave a Comment