Tutorials

When the rules run out, let a small model decide

Sunny Sharma8 min read

In short

Model Router series · Part 4 of 6 · Start from Part 1 · All parts ↓

The rules missed nine cheap questions in Part 3, so here gpt-5.4-nano reads each question and names its type.

  • A classifier reads the need better than rules do. It named the right task type for 24 of 32 held-out questions. The rules got 15. All 68 of its replies were valid JSON.
  • The router costs about $0.000057 and one second per request, and the saving more than covers it. Run for real on the held-out questions, it passed 93 of 96 at $0.00113 per conversation, against $0.00138 for everything on mid. That is 18 percent less.
  • The confidence threshold is a dial, and the safe setting gives the saving back. At 0.7 the estimated cost is $0.00139, no better than having no router. I use 0.0, and I say what that costs.

Where this fits: part 4 of 6. We add a model-based strategy to the gateway from Part 3. The next part protects it from failure.

Paper cut-out transit map with a signal board at the interchange, showing an LLM classifier router choosing a line for each question

Matching words was the wrong tool for those nine questions. A model can read what a question needs, so we let one do it. The catch is that this LLM classifier router now costs money and time on every request. Does it pay for itself?

The LLM classifier router idea

A small, cheap model reads the question first and fills in a small form. Then the real model, picked from that form, answers.

Paper cut-out flow: a classifier call with its cost and time, then the answer call

The form has three fields: the task type, a difficulty, and a confidence between 0 and 1. The prompt lists the five task types with one line each. It tells the model to reply with JSON and nothing else.

CLASSIFIER_PROMPT = """You sort questions sent to an HR helpdesk. Reply with JSON only, no other text:
{"task_type": "...", "difficulty": "...", "confidence": 0.0}

task_type:
- lookup: read one fact from the employee's own records (leave balance, a payslip figure, a holiday date)
- policy: a rule from the company handbook
- reasoning: needs a calculation or comparing several numbers
- action: asks HR to follow up, raise a ticket or change something
- sensitive: about another person's data, a pay dispute, a complaint or a legal matter

difficulty: easy, medium or hard.
confidence: your confidence in task_type, from 0 to 1."""

From router/classifier.py.

We do not trust the reply. A strict schema checks it, and anything that fails falls back to the mid tier.

class ClassifierOutput(BaseModel):
    task_type: Literal["lookup", "policy", "reasoning", "action", "sensitive"]
    difficulty: Literal["easy", "medium", "hard"]
    confidence: float = Field(ge=0.0, le=1.0)

From router/classifier.py.

The router keeps one thing from Part 3: the risk rules still run first. A model should not decide what a policy already decided, and a risk question never pays for a classifier call. Only the rest reach the model.

Turning the form into a tier is a small, pure function, so we can test it, and sweep it later, without calling any model.

def tier_for(c: Classification, tiers: dict[str, str], threshold: float, default_tier: str = "mid") -> tuple[str, str]:
    """Turn a classification into (tier, rule name). Pure, so a threshold sweep needs no model calls."""
    if not c.valid:
        return default_tier, "classifier:invalid"
    if c.task_type == "sensitive":
        return "large", "classifier:sensitive"
    tier = tiers.get(c.task_type, default_tier)
    rule = f"classifier:{c.task_type}"
    if c.difficulty == "hard":
        tier, rule = up(tier), rule + "+hard"
    if c.confidence < threshold:
        tier, rule = up(tier), rule + "+low-confidence"
    return tier, rule

From router/classifier.py.

A sensitive question always goes to large. Otherwise the task type picks a tier from your Part 2 table. A hard difficulty moves the choice up one tier. So does a confidence below the threshold. An invalid reply falls back to mid.

Paper cut-out flow turning the classifier's form into a tier, with the two move-up-one rules

Choose the classifier by measuring

The router brain here is gpt-5.4-nano, because it is the cheapest model we have and it just passed every lookup and policy question. Whether it can also read questions well is a separate fact, so I measured it on the same labelled sample.

python -m scripts.check_p4_classifier
Tuning halfHeld-out half
Rules (Part 3)32 of 3215 of 32
gpt-5.4-nano as classifier27 of 3224 of 32

The rules won on the questions they were written from and lost on the rest. The classifier was steadier: 27 on the tuning half and 24 on the held-out half. As in Part 2, the threshold below was picked on the held-out half, so treat the classifier rows as somewhat optimistic. None of its 68 replies failed validation, so the mid fallback never fired.

The recurring questions came out as you would hope:

QuestionClassifier saysConfidenceTier
Leave balancelookup0.78small
Salary refusalsensitive0.90large
Friday work from homepolicy0.78small
Pay differencereasoning0.74mid

The classifier on its own also calls the salary refusal sensitive, at 0.90. Inside the gateway it never gets the chance, because the risk rule fires first and the classifier is not called.

I tested only gpt-5.4-nano as the brain. If you have an Anthropic key, you can compare claude-haiku-5-5 with the same script (--brains openai:gpt-5.4-nano anthropic:claude-haiku-5-5, after pip install -r requirements-claude.txt). I did not run it, so I make no claim about it.

What the router costs

Every request that reaches the classifier pays for an extra call. Measured: $0.000057 and 1.03 seconds on average. Risk questions skip it.

How many requests must move from mid to small to pay that back? Moving one request from mid to small saves about $0.000987. So the classifier pays for itself once it moves 6 percent of requests down. It moved most of them, as you will see.

The delay is the part to watch. Over a second is added to every request that reaches the classifier, and the model call comes after.

The confidence dial

Below the threshold, the route moves up one tier. Set it high and the router stays cautious, so more questions go to the more expensive tiers. Set it low and the classifier is trusted. The right value is a data question, so sweep it.

Paper cut-out bar chart of cost per conversation at seven confidence thresholds

Sweeping by calling the real models seven times would be slow and costly. Part 2 already ran every held-out question on every tier. If the router sends a question to the small tier, the result is what the small tier did in Part 2. Add the classifier’s own cost and you have an estimate with no new model calls.

ThresholdSmall / mid / large questionsPassed (est.)$ per conversation$ per correct
0.019 / 7 / 693 of 96$0.00118$0.00122
0.517 / 9 / 693 of 96$0.00126$0.00130
0.616 / 10 / 693 of 96$0.00130$0.00134
0.713 / 13 / 693 of 96$0.00139$0.00143
0.81 / 24 / 794 of 96$0.00192$0.00196
0.90 / 21 / 1194 of 96$0.00240$0.00245
0.950 / 19 / 1394 of 96$0.00258$0.00264

Look at what the dial buys. Going from 0.0 to 0.95 raised the cost per conversation by a factor of 2.2 and the pass count by one run. From 0.7 up, the router costs more than not having one: everything on mid is $0.00138.

The jump between 0.7 and 0.8 tells you something. The classifier’s confidence is its own guess, and its values cluster. The four recurring questions came back at 0.74 to 0.9. A threshold of 0.8 moves almost everything up, so the router stops routing. I read that as the model’s confidence being coarse, not a precise measure of whether it is right. That is my reading of this sweep, not something I tested further.

The check recommends the cheapest setting among those within noise of the best, which is 0.0.

At 0.0 no confidence value is below the threshold, so in the setup I ship the confidence field does nothing. Treat it as an experimental signal, not a probability of being right. To test it, group the classifier’s answers by stated confidence and check whether the high groups are right more often. I did not run that.

Run it for real

The sweep is an estimate. The recommended setting then runs through the gateway for real, with the classifier live and every held-out question run three times.

class ClassifierRouter:
    name = "classifier router"

    def __init__(self, classify: Callable[[str, list[str]], Classification], tiers: dict[str, str] | None = None,
                 threshold: float = DEFAULT_THRESHOLD, default_tier: str = "mid",
                 risk_rules: list[Rule] | None = None, name: str | None = None):
        self.classify = classify
        self.tiers = dict(tiers if tiers is not None else load_tiers()[0])
        self.threshold = threshold
        self.default_tier = default_tier
        self.risk_rules = RISK_RULES if risk_rules is None else risk_rules
        self.name = name or self.name

    def __call__(self, message: str, history: list[str] | None = None) -> Route:
        risk = check_risk(message, history, self.risk_rules)
        if risk:
            return Route("large", risk)            # no classifier call, so no overhead
        c = self.classify(message, history or [])
        tier, rule = tier_for(c, self.tiers, self.threshold, self.default_tier)
        return Route(tier, rule, usage=c.usage, seconds=c.seconds)

From router/classifier.py.

SetupPassed$ per conversation$ per correctSeconds
All on mid95 of 96$0.00138$0.001402.4
Best single setting (mid, effort low)96 of 96$0.00138$0.001385.1
Rules router95 of 96$0.00135$0.001372.5
Classifier router, threshold 0.093 of 96$0.00113$0.001173.3

The estimate said $0.00118 and the real run said $0.00113, close enough to trust the sweep. Of the 96 runs, 56 went to small, 23 to mid and 17 to large. The router’s own cost was $0.00005 per conversation, which is inside that total.

Against the baselines: 18 percent less per conversation than everything on mid, and 16 percent less per correct answer than the best single setting. The price is about 0.8 seconds more latency than the rules router, and two fewer passed runs than the mid model.

Two fewer passes is inside the noise rule, but look at which runs they were. All three lost runs are the same question as in Part 2, r-earned-8m, the one about 8 completed months of earned leave. The classifier called it a policy question, which sends it to small, and nano answered “12” from the leave balance without searching the handbook. The router did what it was built to do. The cheap model gave the right number from the wrong source, and the case counts that as a failure.

What this router is not

Agents Part 8 built a supervisor that also used a model to route. It lost. It split the work by domain, and each specialist lacked tools the work needed. This router does something narrower. It picks a model, and the agent keeps every tool whichever model answers. That is why I expected a different result. The numbers do not contradict it: three failed runs against one for mid, all on one question. That is too few questions to call the quality equal.

When not to bother: if your traffic is narrow and the rules already save what you need, stop at Part 3. A classifier adds a call, a second of delay and one more thing to monitor.

Even a router that is right every time has a gap: the model it picks can still be slow, rate limited or down. Right now every route is a single bet.

Next: Part 5: LLM fallbacks: build the safety net before you need it

The series

  1. Not every prompt needs your biggest model
  2. How do you know the small model is good enough?
  3. Your first LLM router is just a few rules
  4. When the rules run out, let a small model decide (this part)
  5. LLM fallbacks: build the safety net before you need it
  6. Prove your LLM router saves money
Sunny Sharma

Security and engineering leader. Notes from running AI, Azure and security in production.

Leave a Comment