Tutorials

Your first LLM router is just a few rules

Sunny Sharma12 min read

In short

Model Router series · Part 3 of 6 · Start from Part 1 · All parts ↓

Part 2 found which questions can go to the small model. This part writes the plain function that sends them there, and measures what it saves.

  • Safety rules go before cost rules. Pay disputes, complaints and other people’s data go to the large model before any cost rule is checked.
  • The rules were perfect on the questions they were written from and wrong on more than half of the rest. They named the right task type for 32 of 32 tuning questions and 15 of 32 held-out ones.
  • The saving is about 2 percent, which is inside the run-to-run noise. The rules router passed 95 of 96 at $0.00135 per conversation against $0.00138 for everything on mid. The rules missed too many cheap questions.

Where this fits: part 3 of 6. We add the first router and the gateway that every later part plugs into.

Paper cut-out transit map with an interchange added, showing a rule-based LLM router sending questions down three lines

The table from Part 2 says which questions can go cheap. Rules are the cheapest way to act on it: no extra model call and no extra wait. So here is the smallest rule-based LLM router that could work, and how far a few rules get you.

What a rule-based LLM router decides

The agent makes several model calls for one question and keeps a conversation going. So the router makes one choice per user turn. It reads the latest message and a little recent history, and picks a tier: small, mid or large.

Note what kind of router this is. The routers in Agents Part 6 pick the next step in a graph. This one picks which model runs. The agent keeps all its tools either way.

The router does not run the model itself. It sits inside a small gateway, the one function the agent calls, so that the later parts can plug in a smarter strategy, a safer executor and a log without touching the agent.

class Gateway:
    def __init__(self, strategy: Callable[[str, list[str]], Route], executor: Callable = single_call,
                 catalog: Catalog | None = None, provider: str = "openai",
                 model_factory: Callable[[str], object] = get_model, sink=None, name: str | None = None):
        self.strategy = strategy
        self.executor = executor
        self.catalog = catalog or load_catalog()
        self.provider = provider
        self.model_factory = model_factory
        self.sink = sink if sink is not None else ListSink()
        self.name = name or getattr(strategy, "name", "gateway")

    def decide(self, messages: list) -> Route:
        latest, history = split_turn(messages)
        started = time.perf_counter()
        try:
            route = self.strategy(latest, history)
        except Exception as exc:  # a broken router must never take the helpdesk down
            return Route(ROUTER_ERROR_TIER, f"router-error:{type(exc).__name__}",
                         seconds=time.perf_counter() - started)
        if route.seconds == 0.0:
            route = Route(route.tier, route.rule, route.usage, time.perf_counter() - started)
        return route

    def invoke(self, messages: list) -> Result:
        """Route one user turn, run it, log it. `messages` is the whole conversation so far."""
        started = time.perf_counter()
        route = self.decide(messages)
        outcome = self.executor(self, route, messages)
        usage = merge_usage(route.usage, outcome.usage)
        cost = usage_cost(usage, self.catalog)
        tools = [c["name"] for m in outcome.messages if m.type == "ai" for c in m.tool_calls]
        record = {"ts": time.time(), "question": split_turn(messages)[0], "gateway": self.name,
                  "tier": route.tier, "rule": route.rule, "router_seconds": route.seconds,
                  "router_cost_usd": usage_cost(route.usage, self.catalog)["cost_usd"] if route.usage else 0.0,
                  "models": list(cost["by_model"]), "attempts": outcome.attempts,
                  "escalated": outcome.escalated, "tools": tools,
                  "input_tokens": cost["input_tokens"], "output_tokens": cost["output_tokens"],
                  "cached_tokens": cost["cached_tokens"], "cost_usd": cost["cost_usd"],
                  "seconds": time.perf_counter() - started, "error": outcome.error}
        self.sink.write(record)
        return Result(outcome.messages, record)

From router/gateway.py.

Paper cut-out diagram of the gateway: strategy, executor and sink, with the part that replaces each

The gateway has three parts. The strategy decides the tier. The executor runs the call. The sink receives one record per request. Parts 3 to 6 each replace one of them. Read invoke: it decides, runs, adds up the cost of the router and the model, and writes a record. If the router itself breaks, the gateway falls back to the mid tier and records router-error, so a bug in the router never takes the helpdesk down.

The hard rule goes first

Before any rule that saves money, we write a rule that protects quality. Pay disputes, grievances, harassment, legal topics, and questions about another person’s data never go to the cheap tier. A cost rule must never override them.

Paper cut-out flow: risk rules first, cost rules second, default mid
RISK_RULES = [
    Rule("risk:dispute", pattern(r"\b(dispute|grievance|complain\w*|harass\w*|legal|lawyer|unfair\w*|"
                                 r"discriminat\w*|wrongful\w*|underpaid|unpaid|not been paid)\b"), "sensitive"),
    Rule("risk:other-person", mentions_someone_else, "sensitive"),
]

From router/rules.py.

The first rule is a pattern for words like dispute, harassment or underpaid. The second looks for another employee’s ID, a possessive name such as “Rohan Mehta’s salary”, or a group such as “what everyone on my team earns”.

This rule is about answer quality on topics where a wrong answer is costly. It is not a security boundary. Who may see whose data has to be decided inside the tools, as Agents Part 10 shows. A regex in front of the model does not stop anyone, and a rephrased question gets past it. Part 2 showed why this matters. In the demo, the leave tool accepts any employee ID the model passes, and nano used it for a colleague’s balance in two of three runs. A router that sends that question to a stronger model improves the odds. It does not close the hole. The repo has a small example of the real fix: scope_to in router/resilience.py wraps the tools so a call for another employee’s ID is refused, and an offline test shows the same refusal whichever tier asks. The models in that test are scripted, so it shows the wiring, and none of the measured runs used it.

There is a fair question here. The refused salary request is the cheapest of the four recurring questions, and the rule sends it to the large model, which costs 3.3 times as much. In Part 2 the mid model passed all 18 sensitive runs, so the evidence does not say the large model is needed. I keep the rule anyway. Those topics are rare, a wrong answer on a harassment complaint costs more than a few cents, and my 6 sensitive questions cannot show the worst case. That is a judgement, not a measurement, and you may decide differently.

Then the cost rules

Each cost rule spots a pattern and names a task type. The tier for that type comes from the break-even table you saved in Part 2.

COST_RULES = [
    Rule("cost:action", pattern(r"\b(ask hr|tell hr|ticket|open a request|escalate|raise a request)\b|follow up with hr"),
         "action"),
    Rule("cost:reasoning", pattern(r"\b(percent(age)?|how much (less|more)|by how much|difference between|"
                                   r"change[d]? from|if i take|how many of them|take home|"
                                   r"why (was|did|is|has) my\b.*\b(lower|higher|less|more|went up|dropped))\b"),
         "reasoning"),
    Rule("cost:lookup", pattern(r"\bhow many (earned |sick |casual )?(leave )?days do i have|"
                                r"\b(earned|sick|casual|leave)\b.*\b(balance|left)\b|\bnet pay in\b|\bwhat was my (net|gross)\b|"
                                r"\bhow much (pf|income tax|tax)\b.*\bin\b|\bshow me my payslip|\bholidays? in\b"),
         "lookup"),
    Rule("cost:policy", pattern(r"\b(policy|policies|can i|carry|notice period|in advance|floating|allowed|"
                                r"eligible|combine|handbook|how long do i have|entitle\w*)\b"), "policy"),
]

From router/rules.py.

Order matters. A ticket request that mentions a number is still a ticket request, so the action rule comes before the reasoning rule. Anything no rule recognises goes to the mid tier, the Part 11 baseline. The router is the first rule that matches, with that safe default.

class RulesRouter:
    """The rules router. Call it with the latest message and the earlier user messages."""

    name = "rules router"

    def __init__(self, tiers: dict[str, str] | None = None, default_tier: str = "mid",
                 risk_rules: list[Rule] | None = None, cost_rules: list[Rule] | None = None):
        self.tiers = dict(tiers if tiers is not None else load_tiers()[0])
        self.default_tier = default_tier
        self.risk_rules = RISK_RULES if risk_rules is None else risk_rules
        self.cost_rules = COST_RULES if cost_rules is None else cost_rules

    def detect(self, message: str, history: list[str] | None = None) -> tuple[str, str | None]:
        """Which rule fires, and the task type it believes in. ("default", None) when nothing matches."""
        history = history or []
        risk = check_risk(message, history, self.risk_rules)   # risk is judged on the recent conversation
        if risk:
            return risk, "sensitive"
        subject = message
        if len(message.split()) <= SHORT_FOLLOW_UP_WORDS and history:
            subject = f"{history[-1]} {message}"
        for rule in self.cost_rules:
            if rule.test(subject):
                return rule.name, rule.task_type
        return "default", None

    def __call__(self, message: str, history: list[str] | None = None) -> Route:
        rule, task_type = self.detect(message, history)
        if rule.startswith("risk:"):
            return Route("large", rule)
        if task_type is None:
            return Route(self.default_tier, rule)
        return Route(self.tiers.get(task_type, self.default_tier), rule)

From router/rules.py.

Short follow-ups such as “and for September?” have no topic of their own, so the router borrows the previous question. Risk is judged on the last two earlier messages plus the new one, so a dispute raised a turn ago does not slip through on a short reply.

I wrote these rules by reading the tuning half of the question set only.

Run it

python -m scripts.check_p3_rules

The first part needs no model and costs nothing, because deciding a route is free with rules. It shows how well the rules read the questions.

tune     rules named the right task type for 32 of 32 questions. Routed small 14, mid 12, large 6.
holdout  rules named the right task type for 15 of 32 questions. Routed small 8, mid 20, large 4.

On the 32 questions I wrote the rules from, they are perfect, because I wrote them from those questions. On 32 questions they have not seen, they name the right task type 15 times. A set of rules fits the wording it was written from. New wording slips past.

A note on method: after writing the rules I checked the held-out agreement once and changed nothing afterwards. If I had edited the rules to fix the held-out misses, the held-out numbers would no longer measure anything.

Here is where the held-out questions landed. Rows are the task type I labelled, columns are the tier the rules chose.

Labelled typeSmallMidLarge
Lookup420
Policy170
Reasoning330
Action060
Sensitive024

Seven of the eight policy questions went to mid, where the nano passes every one. Four of the six sensitive questions reached the large model. Two did not: “I am Rohan’s manager. What is his net pay?” and “My manager has a conflict of interest with a vendor. How do I report it?” matched no risk rule, so they fell through to the default tier.

One misfire is worth knowing about. The rule for other people’s pay matches a capitalised word followed by “earns” or “makes”. That catches “Who earns more, me or Rohan?”, as intended, but it also catches “What makes me eligible for work from home?” and sends it to large. I left the rule as it is, because changing it would change the numbers in this part. Check your own rules against your own questions.

The recurring questions landed like this:

QuestionRuleTier
Leave balancecost:lookupsmall
Salary refusalrisk:other-personlarge
Friday work from homecost:policysmall
Pay differencecost:reasoningmid

What it saved

Then the real run. Each held-out question ran three times through the gateway, and the router’s own cost is zero.

SetupPassed$ per conversation$ per correct answerSeconds
All on mid95 of 96$0.00138$0.001402.4
Best single setting (mid, effort low)96 of 96$0.00138$0.001385.1
Rules router95 of 96$0.00135$0.001372.5

The runs split into small 24, mid 60 and large 12. Quality held: 95 of 96, the same as the baseline, and its one miss was the same r-earned-8m case. Cost per conversation fell 2.3 percent. At 100,000 conversations that is $135 against $138, about $3.

Treat that as close to no saving. In Part 5 the same router ran again with a cascade that never fired, and its cost came out 2 percent lower, with identical routes. A gap that size is run-to-run variation, so 2.3 percent is within noise. The rules sent only a quarter of the runs to the small model, because they did not recognise most of the cheap questions.

The misroutes

The check prints the questions where the router’s choice looks wrong, judged against the Part 2 runs. None were under-served, meaning no question where the router failed and the mid model passed. Twenty-one were over-served: the router sent them above the small tier even though the small model passed most of its runs.

Paper cut-out bars counting the 21 questions sent above the small tier, by the rule that fired

Twelve of those 21 fell through to the default, because no rule matched them. Nine of the twelve were lookup or policy questions in wording the rules did not know:

  • “Which holidays are there in October 2026?”
  • “When are salaries credited?”
  • “Is alcohol reimbursed on client dinners?”
  • “By when are payslips published?”

Five were action questions that the cost:action rule recognised, correctly, and sent to mid, where the table says they belong. The check still counts them, because the small model passed most runs on those cases. Three were risk questions sent to large on purpose, and one was a reasoning question sent to mid, as designed. So not all 21 are mistakes. Nine are clear misses: cheap questions the rules never saw.

A short question can hide a hard problem, and a long one can hide an easy one. Rules see the surface.

When rules are enough

Rules have real strengths. They are fast, free, easy to read and easy to explain to an auditor. If your traffic is narrow and its wording is stable, they may be all you need, and you can stop here.

Try it yourself. Run the check against the tuning half, read its misroutes, add one rule for a pattern you find, and run the held-out check again.

python -m scripts.check_p3_rules --split tune

Write new rules from the tuning half only. Once you look at the held-out misses to write a rule, those questions stop being held out.

If the first line of the output says the router is using built-in defaults, you have not run Part 2 yet. The code does not fail. It falls back to built-in guesses and says so, and your numbers will not match the ones here.

Something has to read the question

The nine clear misses are the interesting part. Nothing in “Which holidays are there in October?” would stop a person routing it to the cheap line. The rules do not see it, because the rules see words. To see the need, something has to read the question.

Next: Part 4: When the rules run out, let a small model decide

The series

  1. Not every prompt needs your biggest model
  2. How do you know the small model is good enough?
  3. Your first LLM router is just a few rules (this part)
  4. When the rules run out, let a small model decide
  5. LLM fallbacks: build the safety net before you need it
  6. Prove your LLM router saves money
Sunny Sharma

Security and engineering leader. Notes from running AI, Azure and security in production.

Leave a Comment