routing simulator
Routing by expected total cost
What it does. Gives each task a level from its wording (tools only say it is coding), reads each model's chance of finishing work at that level from published benchmarks, and sends the task to the model and route with the lowest expected total cost: the try, plus the chance it fails times a person's time and the next attempt.
Where the money comes from. Most tasks don't need the strongest model, and a cheap model that fails costs more than it saves. On the simulator's 2,000 standard tasks, the default spent $0.40 on models per finished task, against $0.73 with Claude Sonnet 5.5 for everything and $1.43 with Claude Opus 5.5, and finished 97.7% of tasks against 97.4% and 97.7%.
When it doesn't help. When most of your work is at the top level, where only the dearest models have a good chance. On coding-only work, cost per success and the default come out about even once failed attempts are counted ($3.00 and $2.93 per finished task). The level rules are word lists, so a listed word can raise a task that isn't hard ("security" in "update the security page's footer"). And the chances come from benchmarks, not yet from your own outcomes.
How it's measured. Offline, by the routing simulator (below). Live, by the "routing" cause in the ledger: the baseline model's cost minus the served model's, at list prices, before the other causes, and by the outcome report.
list-price arithmetic
Prompt caching
What it does. Adds Claude's cache markers when the client set none, sets OpenAI's prompt_cache_key, and never changes the early part of the prompt. On the four Claude backends it asks for a 1-hour cache from a session's second turn.
Where the money comes from. Cached tokens cost a tenth of the input price or less. On a 100,000-token conversation, the hour costs $0.0030 more per turn; after a 20-minute pause the next turn reads the prefix for $0.010 instead of writing it again for $0.25.
When it doesn't help. When the harness already caches well. Claude Code sets its own markers, and caching a client set up itself isn't counted as a Merit Order saving. An agent that never pauses pays the hour's premium for nothing (about 20% more per turn on Sonnet and Opus 5.5).
How it's measured. The "prompt caching" cause, only for caching the gateway added. Checked live: gateway-added Claude caching, turns 2 and later at a tenth of the price. Not yet checked against an Anthropic invoice.
list-price arithmetic
Sticky sessions
What it does. Keeps a session on the route that holds its cache unless moving saves at least 10% plus $0.01 over the rest of the session. Within a task it keeps its model outright.
Where the money comes from. Rebuilding an 80,000-token Claude Opus 5.5 session on a cold backend costs $0.608, against $0.044 for a turn where it is. Stickiness stops a session bouncing between Bedrock and Anthropic.
When it doesn't help. It saves nothing on one-off requests. Sessions that switched backends, and the rebuild cost they paid, aren't reported yet.
How it's measured. Indirectly: it shows up as cache reads in "prompt caching" and as lower costs per turn.
illustration · opt-in
Tool-output compression
What it does. Deterministic rules strip terminal codes and trailing spaces, collapse repeated lines, and keep the first 60% and last 40% of tool output over 24,000 characters. An optional model (LLMLingua-2) drops the least useful words from prose, protecting paths, numbers, identifiers and error lines.
Where the money comes from. A 100,000-character test log becomes about 24,000 characters: roughly 25,000 fewer tokens. On Claude Sonnet 5.5 that is $0.10 less on the turn it arrives and $0.0025 less on every later turn.
When it doesn't help. On short tool output (under 4,000 characters is left alone), and the model-based method removes nearly nothing from code, logs and JSON (5 to 13% of a web page or docs). Whether agents still succeed on compressed results hasn't been measured.
How it's measured. Two causes: "compression (rules)" and "compression (model)".
illustration
Batch prices and GPU headroom
What it does. Accepts OpenAI's Batch API and Anthropic's Message Batches, routes each request like a live one among providers with a batch API and running GPUs with room, then joins the results.
Where the money comes from. Batch APIs cost about half the live price (Moonshot 40% off, xAI 20%). 10,000 summaries of 5,000 tokens in and 500 out on Claude Sonnet 5.5: $150 live, $75 at the batch price.
When it doesn't help. For interactive work, which can't wait up to a day. And you could get the batch discount from the provider directly; the ledger records it as its own cause for that reason.
How it's measured. The "batch discount" cause. Batch APIs are tested against mocks of ten providers, not yet the real ones.
shipped configuration
Your GPUs at break-even
What it does. Starts a replica when the traffic it could serve would cost more than its hourly price plus 20% at API prices for 10 minutes, and drains it when demand falls below break-even. While it runs, the router prices it at 2% of the API price up to 60% load.
Where the money comes from. Steady load is cheaper on a GPU you already pay for. The shipped Qwen3 Coder Next deployment (four H100s, $16 an hour) starts above $19.20 an hour of such work.
When it doesn't help. Today, mostly. On 11 November 2026 spot prices only level-1 work could pay for a GPU, because cheap API models set a high bar. Real GPUs on a real cloud haven't been tested yet.
How it's measured. The "self-hosting" cause; GPU time is compared with cloud bills by the deployment tag.
illustration · opt-in
Response cache
What it does. An exact-match cache of responses, per team or per request, keyed by team so answers never cross teams.
Where the money comes from. A pipeline sending the same $0.02 request 5,000 times a day, with a one-hour TTL, pays about 24 times a day ($0.48) instead of 5,000 ($100).
When it doesn't help. For agents: their turns are almost never repeated. It stores content, so it's off unless a team turns it on.
How it's measured. The "response cache" cause: a hit costs $0 and records the original cost as the saving.
shipped configuration
Budgets and runaway caps
What it does. Monthly budgets for the organization, teams and people, a per-session cap and a per-request ceiling, all checked before a request is sent.
Where the money comes from. Not a saving on any one request: it caps the bad day. A request for 128,000 output tokens on Claude Fable 5.2 with 500,000 tokens in has a worst case of $9.12, under the shipped $25 ceiling. A team's max_session_usd: 20 stops a looping agent at $20, its subagents included.
When it doesn't help. It doesn't make good traffic cheaper.
How it's measured. Refusals say which limit they hit; alerts go out at 80% and 100%.