In short
Model Router series · Part 4 of 6 · Start from Part 1 · All parts ↓
The rules missed nine cheap questions in Part 3, so here
gpt-5.4-nanoreads each question and names its type.
- A classifier reads the need better than rules do. It named the right task type for 24 of 32 held-out questions. The rules got 15. All 68 of its replies were valid JSON.
- The router costs about $0.000057 and one second per request, and the saving more than covers it. Run for real on the held-out questions, it passed 93 of 96 at $0.00113 per conversation, against $0.00138 for everything on mid. That is 18 percent less.
- The confidence threshold is a dial, and the safe setting gives the saving back. At 0.7 the estimated cost is $0.00139, no better than having no router. I use 0.0, and I say what that costs.
Where this fits: part 4 of 6. We add a model-based strategy to the gateway from Part 3. The next part protects it from failure.

Matching words was the wrong tool for those nine questions. A model can read what a question needs, so we let one do it. The catch is that this LLM classifier router now costs money and time on every request. Does it pay for itself?
The LLM classifier router idea
A small, cheap model reads the question first and fills in a small form. Then the real model, picked from that form, answers.

The form has three fields: the task type, a difficulty, and a confidence between 0 and 1. The prompt lists the five task types with one line each. It tells the model to reply with JSON and nothing else.
CLASSIFIER_PROMPT = """You sort questions sent to an HR helpdesk. Reply with JSON only, no other text:
{"task_type": "...", "difficulty": "...", "confidence": 0.0}
task_type:
- lookup: read one fact from the employee's own records (leave balance, a payslip figure, a holiday date)
- policy: a rule from the company handbook
- reasoning: needs a calculation or comparing several numbers
- action: asks HR to follow up, raise a ticket or change something
- sensitive: about another person's data, a pay dispute, a complaint or a legal matter
difficulty: easy, medium or hard.
confidence: your confidence in task_type, from 0 to 1."""From router/classifier.py.
We do not trust the reply. A strict schema checks it, and anything that fails falls back to the mid tier.
class ClassifierOutput(BaseModel):
task_type: Literal["lookup", "policy", "reasoning", "action", "sensitive"]
difficulty: Literal["easy", "medium", "hard"]
confidence: float = Field(ge=0.0, le=1.0)From router/classifier.py.
The router keeps one thing from Part 3: the risk rules still run first. A model should not decide what a policy already decided, and a risk question never pays for a classifier call. Only the rest reach the model.
Turning the form into a tier is a small, pure function, so we can test it, and sweep it later, without calling any model.
def tier_for(c: Classification, tiers: dict[str, str], threshold: float, default_tier: str = "mid") -> tuple[str, str]:
"""Turn a classification into (tier, rule name). Pure, so a threshold sweep needs no model calls."""
if not c.valid:
return default_tier, "classifier:invalid"
if c.task_type == "sensitive":
return "large", "classifier:sensitive"
tier = tiers.get(c.task_type, default_tier)
rule = f"classifier:{c.task_type}"
if c.difficulty == "hard":
tier, rule = up(tier), rule + "+hard"
if c.confidence < threshold:
tier, rule = up(tier), rule + "+low-confidence"
return tier, ruleFrom router/classifier.py.
A sensitive question always goes to large. Otherwise the task type picks a tier from your Part 2 table. A hard difficulty moves the choice up one tier. So does a confidence below the threshold. An invalid reply falls back to mid.

Choose the classifier by measuring
The router brain here is gpt-5.4-nano, because it is the cheapest model we have and it just passed every lookup and policy question. Whether it can also read questions well is a separate fact, so I measured it on the same labelled sample.
python -m scripts.check_p4_classifier| Tuning half | Held-out half | |
|---|---|---|
| Rules (Part 3) | 32 of 32 | 15 of 32 |
gpt-5.4-nano as classifier | 27 of 32 | 24 of 32 |
The rules won on the questions they were written from and lost on the rest. The classifier was steadier: 27 on the tuning half and 24 on the held-out half. As in Part 2, the threshold below was picked on the held-out half, so treat the classifier rows as somewhat optimistic. None of its 68 replies failed validation, so the mid fallback never fired.
The recurring questions came out as you would hope:
| Question | Classifier says | Confidence | Tier |
|---|---|---|---|
| Leave balance | lookup | 0.78 | small |
| Salary refusal | sensitive | 0.90 | large |
| Friday work from home | policy | 0.78 | small |
| Pay difference | reasoning | 0.74 | mid |
The classifier on its own also calls the salary refusal sensitive, at 0.90. Inside the gateway it never gets the chance, because the risk rule fires first and the classifier is not called.
I tested only gpt-5.4-nano as the brain. If you have an Anthropic key, you can compare claude-haiku-5-5 with the same script (--brains openai:gpt-5.4-nano anthropic:claude-haiku-5-5, after pip install -r requirements-claude.txt). I did not run it, so I make no claim about it.
What the router costs
Every request that reaches the classifier pays for an extra call. Measured: $0.000057 and 1.03 seconds on average. Risk questions skip it.
How many requests must move from mid to small to pay that back? Moving one request from mid to small saves about $0.000987. So the classifier pays for itself once it moves 6 percent of requests down. It moved most of them, as you will see.
The delay is the part to watch. Over a second is added to every request that reaches the classifier, and the model call comes after.
The confidence dial
Below the threshold, the route moves up one tier. Set it high and the router stays cautious, so more questions go to the more expensive tiers. Set it low and the classifier is trusted. The right value is a data question, so sweep it.

Sweeping by calling the real models seven times would be slow and costly. Part 2 already ran every held-out question on every tier. If the router sends a question to the small tier, the result is what the small tier did in Part 2. Add the classifier’s own cost and you have an estimate with no new model calls.
| Threshold | Small / mid / large questions | Passed (est.) | $ per conversation | $ per correct |
|---|---|---|---|---|
| 0.0 | 19 / 7 / 6 | 93 of 96 | $0.00118 | $0.00122 |
| 0.5 | 17 / 9 / 6 | 93 of 96 | $0.00126 | $0.00130 |
| 0.6 | 16 / 10 / 6 | 93 of 96 | $0.00130 | $0.00134 |
| 0.7 | 13 / 13 / 6 | 93 of 96 | $0.00139 | $0.00143 |
| 0.8 | 1 / 24 / 7 | 94 of 96 | $0.00192 | $0.00196 |
| 0.9 | 0 / 21 / 11 | 94 of 96 | $0.00240 | $0.00245 |
| 0.95 | 0 / 19 / 13 | 94 of 96 | $0.00258 | $0.00264 |
Look at what the dial buys. Going from 0.0 to 0.95 raised the cost per conversation by a factor of 2.2 and the pass count by one run. From 0.7 up, the router costs more than not having one: everything on mid is $0.00138.
The jump between 0.7 and 0.8 tells you something. The classifier’s confidence is its own guess, and its values cluster. The four recurring questions came back at 0.74 to 0.9. A threshold of 0.8 moves almost everything up, so the router stops routing. I read that as the model’s confidence being coarse, not a precise measure of whether it is right. That is my reading of this sweep, not something I tested further.
The check recommends the cheapest setting among those within noise of the best, which is 0.0.
At 0.0 no confidence value is below the threshold, so in the setup I ship the confidence field does nothing. Treat it as an experimental signal, not a probability of being right. To test it, group the classifier’s answers by stated confidence and check whether the high groups are right more often. I did not run that.
Run it for real
The sweep is an estimate. The recommended setting then runs through the gateway for real, with the classifier live and every held-out question run three times.
class ClassifierRouter:
name = "classifier router"
def __init__(self, classify: Callable[[str, list[str]], Classification], tiers: dict[str, str] | None = None,
threshold: float = DEFAULT_THRESHOLD, default_tier: str = "mid",
risk_rules: list[Rule] | None = None, name: str | None = None):
self.classify = classify
self.tiers = dict(tiers if tiers is not None else load_tiers()[0])
self.threshold = threshold
self.default_tier = default_tier
self.risk_rules = RISK_RULES if risk_rules is None else risk_rules
self.name = name or self.name
def __call__(self, message: str, history: list[str] | None = None) -> Route:
risk = check_risk(message, history, self.risk_rules)
if risk:
return Route("large", risk) # no classifier call, so no overhead
c = self.classify(message, history or [])
tier, rule = tier_for(c, self.tiers, self.threshold, self.default_tier)
return Route(tier, rule, usage=c.usage, seconds=c.seconds)From router/classifier.py.
| Setup | Passed | $ per conversation | $ per correct | Seconds |
|---|---|---|---|---|
| All on mid | 95 of 96 | $0.00138 | $0.00140 | 2.4 |
| Best single setting (mid, effort low) | 96 of 96 | $0.00138 | $0.00138 | 5.1 |
| Rules router | 95 of 96 | $0.00135 | $0.00137 | 2.5 |
| Classifier router, threshold 0.0 | 93 of 96 | $0.00113 | $0.00117 | 3.3 |
The estimate said $0.00118 and the real run said $0.00113, close enough to trust the sweep. Of the 96 runs, 56 went to small, 23 to mid and 17 to large. The router’s own cost was $0.00005 per conversation, which is inside that total.
Against the baselines: 18 percent less per conversation than everything on mid, and 16 percent less per correct answer than the best single setting. The price is about 0.8 seconds more latency than the rules router, and two fewer passed runs than the mid model.
Two fewer passes is inside the noise rule, but look at which runs they were. All three lost runs are the same question as in Part 2, r-earned-8m, the one about 8 completed months of earned leave. The classifier called it a policy question, which sends it to small, and nano answered “12” from the leave balance without searching the handbook. The router did what it was built to do. The cheap model gave the right number from the wrong source, and the case counts that as a failure.
What this router is not
Agents Part 8 built a supervisor that also used a model to route. It lost. It split the work by domain, and each specialist lacked tools the work needed. This router does something narrower. It picks a model, and the agent keeps every tool whichever model answers. That is why I expected a different result. The numbers do not contradict it: three failed runs against one for mid, all on one question. That is too few questions to call the quality equal.
When not to bother: if your traffic is narrow and the rules already save what you need, stop at Part 3. A classifier adds a call, a second of delay and one more thing to monitor.
Even a router that is right every time has a gap: the model it picks can still be slow, rate limited or down. Right now every route is a single bet.
Next: Part 5: LLM fallbacks: build the safety net before you need it
The series
- Not every prompt needs your biggest model
- How do you know the small model is good enough?
- Your first LLM router is just a few rules
- When the rules run out, let a small model decide (this part)
- LLM fallbacks: build the safety net before you need it
- Prove your LLM router saves money