Tutorials

Not every prompt needs your biggest model

Sunny Sharma9 min read

In short

Model Router series · Part 1 of 6 · Start from Part 1 · All parts ↓

An LLM agent usually sends every request to one model, at one price per token, whether the question is easy or hard. This part asks why a conversation costs what it does, and whether we are overpaying for easy questions.

  • Price follows tokens, not difficulty. On gpt-5.4-mini the Friday work-from-home question cost $0.001699 and the leave balance lookup cost $0.001137. Same model, 1.5 times the price.
  • The same four questions cost $44 per 100,000 conversations on gpt-5.4-nano, $138 on gpt-5.4-mini and $458 on gpt-5.4. The big model costs 3.3 times the middle one.
  • A router picks the model before the call, so it can save on every request. It is only worth building if the cheap model answers well enough. We have not shown that yet. Part 2 does.

Where this fits: the first of six hands-on parts. We take the HR helpdesk agent from the Building AI Agents series, cut what each conversation costs, then check whether the answers stayed good. You do not need that series to follow along, because the repo contains the agent.

Paper cut-out transit map with one red line that every HR helpdesk question rides, showing LLM cost per conversation

An LLM agent usually sends every request to one model, and one model has one price per token. That is easy to build and easy to overpay for. The agent in this series is an HR helpdesk for a fictional company, built in the Building AI Agents series. The cost table at the end of Part 11 listed one model for everything, and nobody asked what that did to the LLM cost per conversation. If you run an LLM product and pay for one model on every request, this series is for you.

Employees ask the helpdesk agent about leave, pay and policy. Ask it for your leave balance and it uses gpt-5.4-mini. Ask it why your pay dropped last month and it uses gpt-5.4-mini. The first question needs one lookup. The second needs two lookups and some arithmetic. The model is the same, and so is the price per token.

This series asks one thing. Can we pay less for the easy questions and keep every answer as good as before? Six parts: measure, build a router, make it survive failures, and prove the saving. This part starts with the bill.

Where the LLM cost per conversation comes from

You pay for tokens. Input tokens are everything sent to the model: the system prompt, the tool descriptions, the question and every tool result. Output tokens are what the model writes back. Output costs several times more per token than input. Input that the provider already has in its cache costs a fraction of normal input. Agents Part 11 walks through this in detail, so I will not repeat it.

Paper cut-out cards showing how one conversation's bill is built from input, output and cached tokens

Here are the prices this series uses, as I read them from the vendors’ pages on 18 November 2026. All prices live in one file in the repo, models.yaml, and nothing else in the code holds a price. Prices change, so check them again before you trust any total.

  openai:gpt-5.4-nano:
    input: 0.20
    cached_input: 0.02
    output: 1.25
  openai:gpt-5.4-mini:
    input: 0.75
    cached_input: 0.075
    output: 4.50
  openai:gpt-5.4:
    input: 2.50
    cached_input: 0.25
    output: 15.00

From models.yaml.

Prices are US dollars per million tokens. gpt-5.4-mini is the model the Part 11 agent used, so it is our baseline. gpt-5.4-nano is the cheaper tier and gpt-5.4 the more expensive one. I call them small, mid and large. The repo also lists three Claude models for the optional parts, and the rest of the file maps each tier to a model.

Turning tokens into money is one short function. A routed conversation can touch more than one model, so every model gets its own price.

def usage_cost(usage: dict, catalog: Catalog | None = None) -> dict:
    """Price a conversation.

    `usage` is what LangChain's usage callback collects: one entry per model name, each with
    input_tokens, output_tokens and, when the provider reports it, input_token_details with
    cache_read and cache_creation. input_tokens already includes the cached tokens.
    """
    catalog = catalog or load_catalog()
    total = {"input_tokens": 0, "output_tokens": 0, "cached_tokens": 0, "cost_usd": 0.0, "by_model": {}}
    for model_name, u in usage.items():
        price = catalog.price_for(model_name)
        tokens_in = u.get("input_tokens", 0)
        tokens_out = u.get("output_tokens", 0)
        details = u.get("input_token_details", {}) or {}
        cached = details.get("cache_read", 0)
        written = details.get("cache_creation", 0)
        fresh = tokens_in - cached - written
        cost = (fresh * price.input
                + cached * price.cached_input
                + written * (price.cache_write if price.cache_write is not None else price.input)
                + tokens_out * price.output) / 1e6
        total["input_tokens"] += tokens_in
        total["output_tokens"] += tokens_out
        total["cached_tokens"] += cached
        total["cost_usd"] += cost
        total["by_model"][price.spec] = {"input_tokens": tokens_in, "output_tokens": tokens_out,
                                         "cached_tokens": cached, "cost_usd": cost}
    return total

From router/cost.py.

Four questions, three models

The agent in the repo is the single agent from Agents Part 8 with the extra rules added in Part 9. It has five tools: a handbook search, a leave balance, a holiday list, a payslip lookup and a ticket tool. Its prompt and tools differ from the Part 11 service, so I did not reuse Part 11’s dollar figures. We measure our own baseline.

The four questions come back in every part of the series:

  • Leave balance. “How many earned leave days do I have?” One tool call.
  • Salary refusal. “What is Rohan Mehta’s salary? His ID is E1043.” The agent should refuse and call nothing.
  • Friday work from home. “Can I work from home on Fridays?” A handbook search, then an answer with a citation.
  • Pay difference. “Why was my net pay lower in October than in September 2026?” Two payslip lookups and a comparison.

The first three come from Part 11. The fourth is new, because the first three are all cheap and easy.

Run each one on all three tiers:

python -m scripts.check_p1_baseline

Here is what I got. Each question ran once, on a fresh fake HR system, with the prices above.

Questiongpt-5.4-nanogpt-5.4-minigpt-5.4
Leave balance$0.000311$0.001137$0.003775
Salary refusal$0.000209$0.000550$0.001832
Friday work from home$0.000525$0.001699$0.005635
Pay difference$0.000731$0.002149$0.007075
Average$0.000444$0.001384$0.004579
Per 100,000 conversations$44$138$458

Per question the differences are fractions of a cent. Per 100,000 conversations they are real money, so that is the unit this series uses.

Three things stand out.

First, difficulty shows up as tokens, and tokens are the price. On gpt-5.4-mini the leave lookup read 1,312 tokens and wrote 34. The Friday question read 1,539 and wrote 121. The pay difference read 1,450 and wrote 236, after two tool calls. The salary refusal read 625 and wrote 18, because it needed no tool at all. The model did not charge more for a harder question. It charged for the longer conversation that the question caused.

Second, the tiers are far apart. gpt-5.4 cost 3.3 times gpt-5.4-mini on this mix, and gpt-5.4-nano cost about a third as much. If the nano answers are good enough for even half of the traffic, that is a saving on every one of those requests.

Third, all three models gave a correct answer to all four questions, and all three refused the salary request. I read those four answers myself. That is not a measurement, and it is the reason Part 2 exists.

Cached tokens were zero in every row. Each question ran once on a fresh agent, so there was no earlier prompt to reuse. Part 6 looks at caching again with real conversations.

A router is a signal box

A router sits in front of the models and decides which one a question rides. In a rail network a signal box does not drive trains. It sends each train down the right line. Our router will not answer questions. It will read each one and pick nano, mini or the large model.

Paper cut-out signal box choosing between three model lines with their cost per 100,000 conversations

Two things make that useful. It runs before the model call, so any saving applies to every request that takes the cheap line. And the agent keeps all its tools whichever line it takes. The router changes the model, not the job.

For that to pay off, you need to know what kinds of questions you get. I sorted the helpdesk questions into five task types:

  • Lookup: read one fact from the employee’s own records.
  • Policy: a rule from the handbook.
  • Reasoning: a calculation or a comparison of several numbers.
  • Action: a request for HR to follow up, such as a ticket.
  • Sensitive: another person’s data, a pay dispute, a complaint.

The repo holds a set of 64 labelled questions: 18 from Agents Part 8, 16 from Part 9 and 30 new ones. They split into 12 lookups, 16 policy questions, 12 reasoning, 12 action and 12 sensitive. Half of them are set aside as a held-out half that we will use only for scoring. Part 2 explains why.

One warning. These 64 are a test set I built, with the mix chosen on purpose. They are not a sample of real traffic, and the mix is mine, so do not read the 12 and 16 as “how often employees ask each kind”. Your own mix will differ. Part 6 shows how to log it. Through the series, one conversation means one question run once through the agent, which may take several model calls. Part 6 also replays multi-turn conversations and counts those per turn.

Set it up

You need Python 3.12 or newer and an OpenAI API key. The commands below are for macOS and Linux. The README has the PowerShell and Git Bash versions.

git clone https://github.com/iamsunny/ai-india-2026-model-router-for-cost-latency-and-quality.git && cd ai-india-2026-model-router-for-cost-latency-and-quality
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pytest -q                                   # offline tests, no API key needed
cp .env.example .env                        # then add your OPENAI_API_KEY
python -m helpdesk.ingest                   # builds the handbook index once (one small embedding call)

From README.md.

When the check script finishes, it prints one line per question and model:

gpt-5.4-mini   leave-balance   in  1312 out   34 cached   0 calls 2 $0.001137 2.0s
gpt-5.4-mini   salary-refusal  in   625 out   18 cached   0 calls 1 $0.000550 1.0s

Your token counts will differ a little from mine, because model output varies from run to run. The prices and the shape should match.

If it fails with The handbook index is missing, run python -m helpdesk.ingest once. If you have no key yet, python -m scripts.check_p1_baseline --offline shows the output format with made-up tokens. Do not read anything into those numbers.

What we do not know yet

We believe a small model can handle the easy questions. We have not shown it. If we send a question to a cheaper model and the answer gets worse, we saved money and lost trust. So before we build any router, we need to measure how good each tier really is, task type by task type.

Next: Part 2: How do you know the small model is good enough?

The series

  1. Not every prompt needs your biggest model (this part)
  2. How do you know the small model is good enough?
  3. Your first LLM router is just a few rules
  4. When the rules run out, let a small model decide
  5. LLM fallbacks: build the safety net before you need it
  6. Prove your LLM router saves money
Sunny Sharma

Security and engineering leader. Notes from running AI, Azure and security in production.

Leave a Comment