In short
Model Router series · Part 6 of 6 · Start from Part 1 · All parts ↓
Everything now sits behind one gateway. This part logs every request, replays sample conversations, and ends with one scoreboard for every setup in the series.
- The final setup cost $0.00112 per conversation against $0.00138 for everything on mid. That is 19 percent less, $112 per 100,000 conversations instead of $138. It passed 93 of 96 runs where mid passed 95, a gap inside the noise rule, and it is 1.2 seconds slower.
- Against the best single setting, it is 16 percent cheaper per correct answer and faster.
gpt-5.4-miniat effort low passed 96 of 96 at $0.00138 per correct answer in 5.1 seconds. The router did not beat it on quality. It beat it on cost and speed.- The confidence threshold is where the saving lives. The same setup at 0.7 cost about what no router costs, so the saving was gone.
Where this fits: part 6 of 6. The closing part: one gateway, a request log, a replay, and a drift check you can schedule.

A test run shows the wiring works. The question the series has been building to is bigger: can we run this, watch it, and measure LLM cost savings that hold up?
One door
Everything built so far sits behind one entry point, the gateway from Part 3. The agent calls it and never picks a model itself. The final setup assembles a strategy (rules or classifier), the cascade executor from Part 5, and a log:

def build_final_gateway(strategy: str = "rules", second_provider: bool = False, tiers: dict | None = None,
threshold: float = DEFAULT_THRESHOLD, brain: str = "openai:gpt-5.4-nano", classify=None,
sink=None, model_factory=resilient_factory, catalog=None) -> Gateway:
"""strategy is "rules" or "classifier". `classify` lets a test or an offline run supply the classifier."""
catalog = catalog or load_catalog()
tiers = tiers if tiers is not None else load_tiers()[0]
if strategy == "classifier":
router = ClassifierRouter(classify or LLMClassifier(brain), tiers, threshold=threshold,
name="classifier router")
elif strategy == "rules":
router = RulesRouter(tiers)
else:
raise ValueError(f"strategy must be 'rules' or 'classifier', not {strategy!r}")
executor = CascadeExecutor(second_provider=second_provider, budget=EscalationBudget(max_rate=0.25, window=100))
return Gateway(router, executor=executor, catalog=catalog, model_factory=model_factory, sink=sink,
name=f"final setup ({strategy} + cascade)")From router/final.py.
Because the strategy is a parameter, the same gateway ran every version in this part. The drift check builds its gateway from the same function, so the check and the report measure the same thing.
If it is not logged, the saving is a guess
The gateway writes one record per request. Here is a real one from the replay, a leave-balance question that the classifier sent to small:

{
"ts": 1795178233.0,
"question": "How many earned leave days do I have left?",
"gateway": "final setup (classifier + cascade)",
"tier": "small",
"rule": "classifier:lookup",
"router_seconds": 1.5099101999949198,
"router_cost_usd": 5.675e-05,
"models": [
"openai:gpt-5.4-nano"
],
"attempts": [
{
"spec": "openai:gpt-5.4-nano",
"tier": "small",
"ok": true,
"status": "ok",
"error": null
}
],
"escalated": false,
"tools": [
"get_leave_balance"
],
"input_tokens": 1479,
"output_tokens": 53,
"cached_tokens": 0,
"cost_usd": 0.00036205,
"seconds": 4.809255000000121,
"error": null
}The router chose small by the rule classifier:lookup, and spent $0.0000568 and 1.5 seconds doing so. One model answered, on the first attempt, with one tool call. The whole request cost $0.00036 and took 4.8 seconds. Every number in the rest of this article comes from records like this.
A small module reads them and answers the questions that matter. It never calls a model:
def summarize(records: list[dict]) -> dict:
n = len(records)
cost = sum(r["cost_usd"] for r in records)
secs = [r["seconds"] for r in records]
tokens_in = sum(r["input_tokens"] for r in records)
return {"requests": n, "cost": cost, "cost_per_request": cost / n if n else 0.0,
"per_100k": cost / n * 100_000 if n else 0.0,
"escalation_rate": sum(1 for r in records if r["escalated"]) / n if n else 0.0,
"error_rate": sum(1 for r in records if r["error"]) / n if n else 0.0,
"router_cost": sum(r.get("router_cost_usd", 0.0) for r in records),
"p50": percentile(secs, 50), "p95": percentile(secs, 95),
"input_tokens": tokens_in, "cached_tokens": sum(r["cached_tokens"] for r in records)}From router/telemetry.py.
Measure LLM cost savings with one report
I replayed 10 sample conversations plus the four recurring questions through the final setup, 25 requests in all. The report covers cost by tier and by rule. These are the numbers from the setup I ended on, the classifier at threshold 0.0 with the cascade on top.
25 requests $0.00098 per request about $98 per 100,000
escalated 0% errors 0% latency p50 3.3s p95 4.8s router overhead $0.00126 in total
cached tokens: 1024 of 40847 input tokens
tier requests $/request share of cost escalated p95 s
large 3 0.00198 24% 0% 1.8
mid 6 0.00188 46% 0% 4.6
small 16 0.00045 29% 0% 5.6Read the shares. Sixteen of the 25 requests went to the small tier and carried 29 percent of the cost. Six went to mid and carried 46 percent. Three went to large and carried 24 percent. The router worked, and the bill still comes mostly from the questions that need the bigger models: nine requests, about 70 percent of the cost.
The same log by rule shows which rules earned their place:
rule requests $/request share of cost escalated p95 s
classifier:action 4 0.00173 28% 0% 3.3
classifier:lookup 10 0.00042 17% 0% 4.8
classifier:policy 6 0.00050 12% 0% 5.6
classifier:reasoning 2 0.00219 18% 0% 4.6
risk:dispute 1 0.00229 9% 0% 1.8
risk:other-person 2 0.00182 15% 0% 1.4Caching, checked
Cached tokens reduce the input bill. OpenAI’s caching is on by default. Its guide says a request only hits when it repeats an earlier prefix, and the minimum prefix is about 1,024 tokens, though for models older than the newest ones it varies with settings such as tools and reasoning effort. For the three models here, models.yaml prices a cached token at a tenth of a normal input token (checked 18 Nov 2026). In Part 1, each model call read roughly 600 to 800 tokens (a conversation’s total input divided by its calls), so a single call is usually under the minimum. Only a longer conversation crosses it.
The report agrees. Across the 25 requests, 1,024 of 40,847 input tokens were cached: 2.5 percent. So caching is real here, but small. Any saving from it is a side note, not a plan.
Before and after, on the same questions
The replay compares the final setup with everything on mid, on the same questions. It ran 25 requests: 21 conversation turns plus four recurring questions, which I show one by one. The last row totals the 21 turns, not the four rows above it.
| Question | All on mid | Router | Change | Tier |
|---|---|---|---|---|
| Leave balance | $0.001133 | $0.000365 | -68% | small |
| Salary refusal | $0.000536 | $0.001832 | +242% | large |
| Friday work from home | $0.001648 | $0.000549 | -67% | small |
| Pay difference | $0.002113 | $0.002128 | +1% | mid |
| All 21 turns together | $0.030007 | $0.019541 | -35% | mixed |
Per turn that is $0.001429 before and $0.000931 after, or $143 against $93 per 100,000 turns. Quality on those 21 turns was 20 of 21 before and 20 of 21 after.
The four rows split three ways. The leave balance and the Friday question each got about two thirds cheaper, on the small tier. The salary refusal got 3.4 times more expensive, on purpose, because the risk rule sends it to large. The pay difference needed the mid model and stayed there.
Do not read 35 percent as the saving. These 21 turns are a sample of conversations I wrote, with a lot of lookups in them. The saving depends on the mix of questions. The scoreboard below is the measure, on the held-out questions.
The scoreboard
Every row is measured on the same 32 held-out questions, 3 runs each, 96 runs. Costs are per conversation.
| Setup | Part | Passed | $ per conversation | $ per correct | Seconds |
|---|---|---|---|---|---|
All on large (gpt-5.4) | 2 | 88 of 96 | $0.00438 | $0.00478 | 2.9 |
All on mid (gpt-5.4-mini), the baseline | 2 | 95 of 96 | $0.00138 | $0.00140 | 2.4 |
| Best single setting (mid, effort low) | 2 | 96 of 96 | $0.00138 | $0.00138 | 5.1 |
| Rules router | 3 | 95 of 96 | $0.00135 | $0.00137 | 2.5 |
| Classifier router, threshold 0.0 | 4 | 93 of 96 | $0.00113 | $0.00117 | 3.3 |
| Rules router + cascade | 5 | 95 of 96 | $0.00132 | $0.00134 | 2.4 |
| Final setup: classifier + cascade, threshold 0.0 | 6 | 93 of 96 | $0.00112 | $0.00116 | 3.6 |
Against the baseline, the final setup costs 19 percent less per conversation. Per 100,000 conversations that is $112 instead of $138, a saving of $26. Against the best single setting it costs 16 percent less per correct answer. It is 1.2 seconds slower than everything on mid and 1.5 seconds faster than effort low.
What this does and does not show: the router passed 93 runs against 95 for the mid model and 96 for effort low. A gap of 2 or 3 runs is under the noise rule, so I cannot say quality dropped, and I cannot say it did not. In the Part 4 run of the same router, all three lost runs were r-earned-8m, where nano gave the right number without searching the handbook, and the cascade cannot see that. If an answer from the wrong source costs you more than the saving, use effort low and skip the router.
Read this table as exploratory. I picked the break-even table, the threshold and the best single setting by looking at these same 32 questions, so the router rows are optimistic. The gap to the baseline is also smaller than three runs suggest: it is one question out of 32, failed three times. The mid model failed that same question once in three runs, and three times in three at effort none. One question cannot show that quality is equal, and it cannot show that it is worse. To claim more, freeze the setup and score it once on questions nobody has looked at. I have not done that.
One more row belongs on the scoreboard: everything on gpt-5.4-nano passed 88 of 96 at $0.00040 per conversation ($0.00043 per correct answer). The reason not to use it is the kind of failure, not the count. Two of its failed runs listed a colleague’s leave balance, which Part 2 shows.
The saving also depends on the mix. In the Part 4 run of this router (the final setup adds only a cascade that never fired), 17 of the 96 runs (18 percent) went to the large tier, and they made 43.8 percent of the cost. By task type, against everything on mid, cost per conversation moved about like this: lookups down 68 percent, policy questions down 63, reasoning down 58 (the classifier labelled most reasoning questions as lookups or policy), actions down 4, and sensitive questions up 216 percent, about 3.2 times. If those 17 runs had stayed on mid, the cost would have been about $0.000785, a saving of roughly 43 percent. So on this mix the risk rule costs about $34 per 100,000 conversations, and the router saves about $25 to $26. I would still keep the rule. The point is that your saving is a function of your mix.
Now the dial. I ran the final setup three ways. They are separate runs, so differences of a run or two are noise. The first row is a second run of the rules router plus cascade, which scored 95 of 96 in Part 5. Two runs of the same setup gave 95 and 93, which shows the noise directly.
| Setup | Passed | $ per conversation |
|---|---|---|
| Rules + cascade | 93 of 96 | $0.00131 |
| Classifier at threshold 0.7 (the default) + cascade | 94 of 96 | $0.00142 |
| Classifier at threshold 0.0 + cascade | 93 of 96 | $0.00112 |
At 0.7, the router sent more questions up a tier. Quality was a run better, and the cost was about the same as no router, so the saving was gone. The cost at 0.7 is also unstable. It was $0.00142 here, $0.00169 in the drift example below, and my Part 4 estimate was $0.00139. I think the classifier’s confidence values sit close together near 0.74 to 0.78, so a small shift moves many questions across the line. Same code, a different number, a different result. A router’s saving lives in the threshold, so measure at the setting you will ship.
Run it
python -m scripts.check_p6_report --strategy classifier --threshold 0.0It replays the conversations, prints the report and the before and after table, runs the held-out set for the scoreboard row, and saves a baseline file for the drift check. Allow 10 to 15 minutes and under a dollar. If you start with --quick, treat the numbers as a wiring check only.
Drift
A router is never finished. Prices change, vendors update the model behind a name, and your users ask new things. The drift check re-runs the held-out set against the baseline you saved and fails on three conditions.

def gate(current: dict, baseline: dict, runs_per_case: int = 3) -> list[str]:
"""Return the reasons the router fails the gate, or an empty list. Both dicts have passed, runs,
cost_per_conversation and escalation_rate."""
failures = []
lost = baseline["passed"] / baseline["runs"] - current["passed"] / current["runs"]
if lost * current["runs"] >= NOISE_CASES * runs_per_case:
failures.append(f"quality dropped: {current['passed']} of {current['runs']} passed, "
f"was {baseline['passed']} of {baseline['runs']}")
base_cost = baseline["cost_per_conversation"]
if base_cost and current["cost_per_conversation"] > base_cost * (1 + MAX_COST_INCREASE):
failures.append(f"cost per conversation rose from ${base_cost:.5f} to ${current['cost_per_conversation']:.5f}")
if current["escalation_rate"] - baseline["escalation_rate"] > MAX_ESCALATION_INCREASE:
failures.append(f"escalation rate rose from {baseline['escalation_rate']:.0%} to {current['escalation_rate']:.0%}")
return failuresFrom router/drift.py.
A quality drop of 2 cases or more fails, the same noise rule as before. Cost per conversation up more than 20 percent fails. An escalation rate up more than 10 points fails. The last is the early warning: a cheap model that quietly gets worse starts failing the cascade check before anyone complains. The 20 percent cost gate is wider than the 19 percent saving, so a router could lose its whole saving before the gate fires. A tighter gate, such as 10 percent, suits a setup like this one. Two more warnings come with it, for prices older than 30 days and for a model with no price in models.yaml.
python -m scripts.drift_checkIt exits with 0 on a pass and 1 on a fail, so it fits in a scheduler. When it fails, look at the failure line and re-measure with Parts 2 and 4 before you touch the router.
My first run of it failed, and the failure was my bug, not drift:
baseline (2026-11-20): 93 of 96 passed, $0.00112 per conversation, escalated 0%
now: 95 of 96 passed, $0.00169 per conversation, escalated 0%
FAIL
- cost per conversation rose from $0.00112 to $0.00169The baseline came from the classifier at threshold 0.0. The check rebuilt the gateway at the default 0.7, so it compared two different setups, and 0.7 costs more. The baseline file now stores the threshold, the check reads it back, and a test covers it. The same run after the fix:
baseline (2026-11-20): 93 of 96 passed, $0.00112 per conversation, escalated 0%
now: 93 of 96 passed, $0.00113 per conversation, escalated 0%
PASSWhatever the baseline was measured with must be saved next to it, or the check will compare the wrong thing and look like drift.
What a router cannot fix
A router changes which model answers. It does not touch the rest of the bill. Take the first leave-balance question on gpt-5.4-mini in Part 1: 1,312 input tokens and 34 output tokens. At the prices in models.yaml, 87 percent of that $0.001137 is input. A one-line question carries a system prompt, five tool descriptions and the tool results, and routing leaves all of it as it was.
A shorter prompt, trimmed tool descriptions or tighter retrieval might cut cost more than any router. I did not test that, so I will not claim it. I will say where I would look first.
That is the argument of the whole series. Measure first, because the average hides the cases that matter. Start with rules if they hold on questions they were not written from. Let a classifier read the question if they do not, and check that it pays for itself. Add a safety net with a limit on every layer. Then log, replay and gate, so the number stays true. If you would only run one thing, run the held-out comparison on your own questions, and ship a router only when the saving beats the noise and the failures it adds are ones you can live with.
Everything here lives in the repo: model-router. This series is the hands-on companion to my AI-India 2026 talk in Pune.