Merit Order
GitHub

How it works

One request, one session, one month.

Follow a coding agent's request through the gateway. Then watch what changes when it becomes a conversation, and what a month of traffic adds. Worked examples are list-price arithmetic on the shipped prices of 12 November 2026; routing's results come from the routing simulator.

Part 1

A request

Your tool sends model: "auto" to the gateway, with a Merit Order key. The tool doesn't change. Everything below happens before the first byte goes to a provider, in well under the time a model takes to answer.

The path of one request: Arrives: Anthropic or OpenAI format, any tool; Checks limits: budgets, rate limits, caps; refused before sending; Gets a level: tools say coding; the wording sets the level; Finds who may serve it: room in the window, allowed, chance above the floor; Prices every route: rebuild + turns × cost per turn; Picks the lowest expected total: try + (1 − chance) × (person + retry); Shapes and sends: cache markers, compression, effort; fails over; Records it: ledger row: cost, baseline, savings by cause, reason.01ArrivesAnthropic or OpenAI format, any tool02Checks limitsbudgets, rate limits, caps; refused before sending03Gets a leveltools say coding; the wording sets the level04Finds who may serve itroom in the window, allowed, chance above the floor05Prices every routerebuild + turns × cost per turn06Picks the lowest expected totaltry + (1 − chance) × (person + retry)07Shapes and sendscache markers, compression, effort; fails over08Records itledger row: cost, baseline, savings by cause, reason

1. Limits first

Organization, team, person and session budgets, rate limits and the per-request cost ceiling are checked from the gateway's own counters. Over a limit, the request is refused before it costs anything, and the error says which limit.

2. A level for the work

Rules read the request, with no model call. Plain text is general 2. Tools or code make it coding, at 3, and the task's wording then sets the level: "write tests", renames, typos and docstrings put it at 2; hard wording such as "race condition", "migration" or "under load" adds one for each kind, up to two; "architecture" or "design a" puts it at 5. What the harness adds (Claude Code's system reminders, Codex's AGENTS.md) isn't read as the task. Length never raises a level.

3. Each model's chance

Every model has a 1 to 5 score per kind of work, from Epoch AI's benchmark data against a fixed bar, or from your own evals, and from it a chance of finishing work at each level. A model may serve the task when the conversation fits its context window, the team may use it, and its chance is above the floor.

4. Price for the rest of the session

Every route of every model that may serve the task is priced as rebuild + expected turns × cost per turn. The rebuild is what it costs to get the conversation into that provider's cache: zero where it is already warm. That is the try.

5. Pick, shape, send, record

The lowest expected total wins: try + (1 − chance) × (person's time + next attempt), with a person's time at $1 a failure by default (a minute of a developer), set per team. The gateway adds cache markers, compresses tool output if the team turned it on, and sets reasoning effort from the level. If the provider fails, the next route takes it, before anything reaches your tool. A ledger row records the cost, what the baseline model would have cost, and why.

Simulated

What the default changes

On the routing simulator's 2,000 standard tasks (every model call mocked, each failed attempt priced at $15 of someone's time), expected total cost finished 97.7% of tasks at $0.40 of model spend each, against the old threshold's 97.5% at $1.01; $3.08 against $3.63 in total, counting failed attempts. Cost per success (model_choice: cost_per_success) spends less on models, $0.40, but finishes 96.8%. Escalating to a stronger model inside a task (routing.cascade, off by default) finished 97.8% but cost $3.08 in total, counting failed attempts, so it stays opt-in. The chances come from benchmark scores; they haven't been checked against real outcomes yet. All the results.

Option

Name a model

A request for claude-opus always gets Claude Opus. Merit Order then only picks the cheapest of its routes (Anthropic, Bedrock, Vertex AI, Foundry, OpenRouter) with the same stickiness and failover.

Part 2

A session

An agent resends the whole conversation every turn. Providers charge a tenth of the input price or less for the part they still have cached. So where a conversation lives matters more than any single request.

Four later turns of a conversation already 100,000 tokens long, on Claude Sonnet 5.5. Each turn reads the cached prefix and writes about 2,000 new tokens. After a 20-minute pause a 1-hour cache is still there, so the turn costs $0.010 to read; with a 5-minute cache the whole prefix is written again for $0.25.turnprefix cost21$0.01022$0.01023$0.010a 20-minute pause24$0.0101-hour cache: still there, read it$0.255-minute cache: gone, write it all again
Turns 21 to 24 of a long session. Grey: read from cache. Indigo: the new tokens of each turn. List-price arithmetic on Claude Sonnet 5.5.

Sessions stay where their cache is

Moving a session means paying again for the whole context. An 80,000-token Claude Opus 5.5 session would cost $0.608 to rebuild on a cold backend, against $0.044 for a turn where it is. So a session moves only if the move saves at least 10% plus $0.01 over the rest of the session, and within a task it keeps its model outright.

Longer caches where pauses pay for them

On the four Claude backends, the gateway asks for a 5-minute cache on a session's first turn and an hour from its second. The hour costs $0.0030 more per turn on a 100,000-token conversation. A person stepping away for 20 minutes then saves a full rewrite. The router counts each team's real share of such pauses when it compares routes. Claude Code sets its own markers, and those go as they are.

Compression that keeps the cache valid

Turned on per team, the rules strip terminal codes, collapse repeated lines and keep the head and tail of tool output over 24,000 characters. They are deterministic, and the model-based method stores its cut once, so every later turn sends the same bytes and the cache still hits.

When a conversation outgrows its model

Length never raises the level. It only rules out models whose window can't hold the conversation plus the reply. A session that outgrows its model moves to one with room at the same level, paying one rebuild. Subagents keep their own routing state, so they don't knock the main conversation off its cache.

Part 3

A month

Over a month, the gateway sees enough traffic to decide when your own GPUs pay, to send work that can wait to batch prices, to hold every team to a budget, and to learn whether its levels are right.

GPUs

Started at break-even, priced as nearly free while they run

The price the router puts on a running GPU: 2% of the API price up to 60% load, then rising with the square of the load to the full API price at 100%.100%50%00% load60%100%2% of API price
Price of a token on a running replica, as a share of the API price.

Every minute the controller asks what the traffic a model could serve costs at API prices. When that beats the replica's hourly price plus 20% for 10 minutes straight, a replica starts. The shipped gpu-qwen-coder deployment, Qwen3 Coder Next on four H100s at $16 an hour, starts above $19.20 an hour of such work. It drains when demand stays below break-even for 10 minutes after at least 30 minutes of serving.

Honest about the bar: on 11 November 2026 spot prices, only level-1 work could pay for a GPU, for example gpt-oss-20b on one L4 from about 72,000 tokens a minute. Models → GPU break-even in the console shows this for your traffic.

Batch

Work that can wait

OpenAI's Batch API and Anthropic's Message Batches go through the same router, among providers with a batch API (about half price; Moonshot 40% off, xAI 20%) and running GPUs with room. GPU work runs at low priority in spare capacity only and moves to a provider's batch API if it hasn't run in 12 hours.

Budgets

Refused before it's spent

Monthly budgets for the organization, each team and each person, a GPU budget, rate limits, a per-session cap and a per-request ceiling. All are checked before a request is sent. Alerts go out at 80% and 100%. A looping agent at 3 a.m. gets its next request refused, not an email hours later.

Ledger

Savings by cause, checked against bills

Every request gets a row: actual cost, the baseline model's cost, and savings split into seven causes that add up. Every 6 hours the worker reads each provider's own cost data (Anthropic, OpenAI, AWS, Google Cloud, Azure) and alerts when a settled day differs by at least $1 and 5%. So far only OpenRouter's costs have been compared with a real bill; they matched for every request.

Learning

Your own traffic as the benchmark

The learning loop: exploration routes about 4% of task starts one level up or down; outcome signals record what followed; Reports, Levels compares the arms once each has 200 task starts; an admin applies the recommendation as routing.level_map.Exploration~4% of task starts,one level up or downOutcome signalstests, builds, fix loops,resends, correctionsReports → Levelseach arm's success,after 200 task startsApplyrouting.level_map,by an adminthe next month starts from the corrected levels

All opt-in per team. Exploration never touches a named model, a client's X-Route-Floor or a nearly spent budget, and only starts a task whose cache is cold. A report line reads like "coding level 3: level 2 does as well in 98%; consider lowering". A learned router can also correct levels from outcomes; it proves itself in shadow mode first and routes only when at least 80% sure. All of this is tested against mocks and synthetic traffic, not yet real traffic.

Pre-production: version 0.0.1

Most of Merit Order is tested only against mocks of each provider's API. Checked live so far: OpenRouter, Anthropic and OpenAI, Claude Code sessions through the gateway, the Docker Compose stack, and GPUs on a local kind cluster with a stand-in vLLM.

Not yet checked live: the other providers, real GPUs, a real Codex CLI session, Entra and Google sign-in, batch APIs and bills. Set an organization budget before you send it real traffic. What's verified.

Next: where the savings come from, and a calculator, or the full story in the docs, starting with How it all works.