Routing by task complexity: what the 2026 evidence actually shows

Between June 26 and July 29, 2026, ten model-routing products launched on Hacker News. The two biggest threads pulled 216 and 123 points, and the arguments in both landed on the same objection within a dozen comments. Meanwhile the blog layer settled on a number — “routing cuts your bill 40-85%” — that traces back to a 2024 paper about models nobody runs anymore. This post is the arithmetic underneath the argument, computed on rates verified today. The conclusion is not the one either camp expects.

Why this got loud in the last six weeks

Two things happened at once.

The first is supply. Routing went from a research topic to a product category in about five weeks. The Hacker News record, by date and score:

The second is that the economics moved underneath everyone. On July 30, 2026, OpenAI repriced the GPT-5.6 family: Luna fell 80% to $0.20/$1.20, Terra fell 20% to $2/$12, and Sol was left untouched at $5/$30.

That asymmetry matters more than the discount. Before the cut, the cheapest and dearest tier of one model family were 5x apart. After it, they are 25x apart.

A price cut that hits one tier and not another is a routing event. Every threshold anybody tuned in June was tuned against a 5x gap, and nothing in their code changed when the gap became 25x.

Four different things called “routing”

Most of the confusion in these threads comes from four unrelated architectures sharing a word. They have different costs, different failure modes, and different evidence behind them. Separating them is half the value of thinking about this at all.

1. Predictive routing. A classifier reads the prompt before any model sees it and picks a destination. This is what OpenRouter’s Auto does, what Bedrock’s Intelligent Prompt Routing does, and what most of the June-July launches do. One decision, no second call, and the whole thing rests on the classifier being right.

2. Cascading with escalation. Send everything to the cheap model. Judge the answer. Escalate the failures. No classifier required; you pay for some work twice. This is the oldest pattern in the space and, as the arithmetic below shows, the one with the most forgiving economics.

3. Vendor tier aliases. The gpt-5.6 alias resolves to gpt-5.6-sol. This is naming, not routing — OpenAI’s model guidance is explicit that you “choose the target model for the workload” yourself. Worth stating because “the API routes for me” appears in these discussions and is not true of the OpenAI alias.

4. The in-model dial. Anthropic’s effort, OpenAI’s reasoning.effort, Google’s thinking_level. Same model, different amount of work. No routing infrastructure at all. We wrote about this dial in June and argued it had made external routers obsolete. That claim needs revisiting, and it gets its own section below.

Anything a vendor or a blog says about “routing” applies to exactly one of these. Most published savings figures come from category 2 and are quoted in defence of category 1.

What the evidence says, as opposed to the blogs

The marketing consensus is remarkably uniform: 40-85% savings, 95% of frontier quality. Chase the citations and they converge on RouteLLM, which measured a strong-versus-weak gap in the GPT-4 and Mixtral era. It was a good result in 2024. It is being quoted in 2026 as though the model landscape had not turned over twice since.

The current picture is less comfortable.

LLMRouterBench (January 12, 2026) unified the prior router benchmarks into 400,000+ instances across 21 datasets and 33 models. Its headline finding is that many routing methods perform similarly under unified evaluation, and that several recent approaches — commercial routers included — fail to reliably outperform a simple baseline. A second finding is quieter and more useful: larger model ensembles show diminishing returns compared to careful model curation. More destinations is not better routing.

Small language models as front-door routers (March 26, 2026) asked whether a 1-4B model can do the classification cheaply enough to be free. On cost and latency it can: the best candidate, Qwen-2.5-3B, hit 988 ms median at zero marginal cost. On accuracy it could not. Best exact-match was 0.783, and the study’s own viability bar — 0.85 accuracy at 2,000 ms P95 — was met by none of the models tested. The authors name the remaining gap as 6-8 percentage points of accuracy, plus an unresolved question about whether correct classification even improves the final output.

A survey of dynamic routing and cascading (Trinity College Dublin, revised April 21, 2026) catalogues seven routing paradigms and declines to crown one, concluding that open challenges remain in making routing generalize across architectures and applications.

Read together: routing works, but the classification step is the weak link, and the products built on it have not demonstrated an edge over simple assignment.

The vendor router that stopped moving

Amazon Bedrock’s Intelligent Prompt Routing has been generally available since April 2025 and is the closest thing to a default in enterprise stacks. Its supported-model table, in the live documentation, reads: Claude 3 Haiku, Claude 3.5 Haiku, Claude 3.5 Sonnet (v1 and v2), Llama 3.1 8B and 70B, Llama 3.2 11B and 90B, Llama 3.3 70B, Nova Lite and Nova Pro.

That is a roster from 2024 and early 2025. The documented constraints are equally telling: it is “only optimized for English prompts”, it “can’t adjust routing decisions or responses based on application-specific performance data”, and a router must be configured with exactly two models within the same family.

In fairness, the same page’s CLI example passes anthropic.claude-sonnet-4-5 as a model ARN, so the support table and the example contradict each other and configured routers likely reach further than the table admits. Either way the point stands: the managed option optimizes across a model set at least a generation behind the one you would deploy today, and picking correctly between two obsolete models is worth less than picking the right modern one.

The arithmetic nobody in the threads did

Here is the thing that reframes the whole debate.

Take a cascade: try the cheap model, and if the answer fails your check, redo it on the flagship. Let c be the cheap model’s cost for the task, C the flagship’s, and f the fraction of tasks the cheap model fails.

• Always flagship: C
• Cascade: c + f x C (you always pay the cheap call; a fraction of the time you also pay the full one)

The cascade wins when c + f x C < C, which rearranges to:

f < 1 - c/C

That is the entire economics of task-complexity routing in one line. Note what it does not contain: any term for how clever the classifier is.

Now put real numbers in it. These are effective blended rates — 92% input / 8% output at an 82% cache hit rate, the same assumption behind every “effective” figure on this site, computed from the pricing snapshot rather than typed by hand:

Read the right-hand column slowly. Routing everything to Luna first and escalating to Sol is cheaper than going straight to Sol even if Luna fails 95% of the time. A cheap-first cascade into a 143x-cheaper Chinese model survives a 99% failure rate on cost grounds alone.

This is why the router benchmarks find so little separation between methods. The cost surface is nearly flat with respect to classifier quality. A router that is 78% accurate and a router that is 92% accurate produce almost identical bills, because both sit far inside a tolerance band that runs to 80-99%.

That reframing changes what you should build. If you have 80% headroom, do not spend engineering effort on a smarter classifier. Spend it on the check that decides whether to escalate: schema validation, a test suite, a retrieval-grounded assertion, a cheap second opinion, a human review queue for the 2% that matter. The classifier is the part with published diminishing returns. The detector is the part with none.

The cache trap: where routing actually loses money

Now the part the Hacker News threads got right by instinct and nobody costed.

The first substantive objection in the 216-point routing thread was cache invalidation, and it arrived within a handful of comments. One commenter put the mechanism plainly:

“If you statelessly route each new request: yes it does end up being more expensive!”

— ai_slop_hater, Hacker News, June 26, 2026

In the local-versus-hosted thread two days later, the same worry showed up as a continuity problem rather than a billing one:

“There’s a hidden tax with routing this way, the original model loses context of what was done and either performs a regression or hallucinates.”

— hmokiguess, Hacker News, June 28, 2026

Both threads circle the same thing for dozens of comments without a number attached. Here is the number.

Take one turn of a running agent: a 50,000-token prefix (system prompt, tool definitions, conversation history) and 500 tokens of output. The session has been going for a while, so on the incumbent model that prefix is cached.

Routing that turn from a warm Opus 5 to a cold Haiku 4.5 — from the most expensive model in the lineup to one costing a fifth as much — makes the turn 40% more expensive.

The mechanism is a price inversion that the list prices hide completely: an Opus 5 cache read costs $0.50/M and a fresh Haiku 4.5 input token costs $1.00/M. The flagship’s cached token is half the price of the cheap model’s fresh one. Once a prefix is warm, the cheap model is not competing against $5/M. It is competing against $0.50/M, and it loses on the input line.

The exact crossover, holding the workload fixed: above a 94.8% cache hit rate on Opus 5, the cold Haiku turn is always dearer. Long-running agents with a stable prefix live right at that boundary.

And the third row is worse than either. If the detour outlives the five-minute TTL, the next Opus turn does not resume at $0.50/M — it pays a cache write at $6.25/M, 12.5x the read rate, which on this prefix is $0.325 against $0.0375. One badly-timed reroute can cost more than eight turns of the thing you were trying to avoid.

Two independent confirmations that this is not a theoretical concern.

OpenRouter built the fix into its product. Auto classifies a prompt into roughly 30 task types and ranks candidate models on seven days of aggregate community spend, with a cost_quality_tradeoff dial from 0 to 10. But subsequent requests in a conversation pin to the same model and provider, via fingerprinting or an explicit session_id, in the docs’ own words to maximize prompt cache efficiency. The most-used router in the ecosystem deliberately stops routing once a session is warm.

Anthropic documents the same constraint for its in-model dial. From the effort documentation: because effort shapes the rendered prompt, changing it between requests does not preserve cached prefixes from earlier turns, and the stated best practice is to hold effort constant within cached conversations and vary it across workloads instead.

That is the same rule, arrived at from the opposite direction. Even the zero-infrastructure version of routing — one parameter on one model — has to be set per workload rather than per turn, for exactly the cache reason. If the cheapest possible switch is still too expensive to make mid-conversation, a model switch certainly is.

The tokenizer skew: $/token is not $/text

A second distortion sits under every model comparison, and it got worse in 2026.

Anthropic’s pricing documentation states that Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text, while Sonnet 4.6 and earlier use the previous one. Both tokenizers are in the current lineup simultaneously.

So compare Claude Haiku 4.5 (old tokenizer, $1/M input) against Claude Opus 5 (new tokenizer, $5/M input). The list ratio is 5x. Feed both the same document and the ratio of what you are billed is closer to 6.5x, because the flagship counts the same text as ~30% more tokens before any rate is applied.

The direction is worth noting: it makes the downgrade more attractive than the price sheet suggests, not less. Sonnet 5 against Haiku 4.5 goes from a 2x list ratio to about 2.6x per unit of text. But the sign flips whenever you compare in the other direction, and either way a threshold tuned on list prices is tuned on the wrong number.

Across vendors the effect is smaller than people assume for prose and larger for the content agents actually move. Our own tokenizer measurements (fixed English corpus, each vendor’s published tokenizer, June 10, 2026) put chars-per-token on prose at 5.28 for OpenAI, 5.259 for DeepSeek and 5.228 for Qwen — within about 2%. On code and JSON the spread widens to nearly 28% between the extremes. If you are routing chat, ignore this. If you are routing an agent that ships diffs and tool results, it is a real term.

A smaller cousin of the same effect: the tool-use system prompt Anthropic injects differs by model — 286/406 tokens on Opus 5, 496/588 on Haiku 4.5, 675/804 on Opus 4.7. A router swaps that invisible overhead too, and on short calls it is not noise.

The vendors disagree with each other

Now the section that revises something we published.

In June we argued in The reasoning dial nobody touches that the external router was a pre-reasoning-era pattern — that you no longer need to cascade between a smart model and a dumb one, because the same task decomposition now lives inside one model as an effort parameter.

Anthropic’s documentation agrees with that almost word for word. Effort is “the primary control for trading off intelligence, latency, and cost” on Claude Fable 5, and the Opus 5 guidance says to use low and medium “liberally as your primary control for token cost and response time.”

OpenAI’s documentation says the opposite. Its reasoning guide instructs you to “treat reasoning.effort as a tuning knob, not the primary way to recover quality,” and directs you to the model tier instead: start with gpt-5.6, step up to gpt-5.6-sol for the hardest problems, step down to terra or luna for lower cost.

Both are defensible, and the difference is a product of lineup shape, not disagreement about physics:

• Anthropic’s tiers are 5x apart on effective rate (Haiku 4.5 to Opus 5) and share an architecture and a effort implementation that spans five levels. Moving the dial is cheap and moving the model is a bigger jump than most tasks need.
• OpenAI’s tiers are now 25x apart after the July 30 cut. When the tier below is 25x cheaper, no amount of effort tuning on the expensive tier competes with just using the cheap one.

So the honest version of our June claim is narrower than we wrote it. The in-model dial did absorb the routing problem — on lineups where the tiers are close together. Where the vendor has opened a 25x gap inside one family, model selection is back to being the first-order decision and the vendor says so itself. We generalized one vendor’s philosophy into a general rule. It is not one.

The durable statement is the ordering rule, not the winner: reach for the cheapest control that moves cost by the amount you need. That is the effort dial when the gap you need to close is 2x, and the model tier when it is 25x. The gap is a property of the price sheet, which changes without warning — see the next section.

Local models are a different equation entirely

The second-biggest thread of the period was about routing between a local model and a hosted one, and it deserves a caveat rather than inclusion, because the formula above does not survive the move.

Everything here prices marginal tokens. Local inference has no marginal token price. Once the hardware exists and is idle, c is approximately zero, and f < 1 - c/C degenerates to “always try local first” for any failure rate below 100%. Which is correct as far as it goes — and useless, because the real costs moved somewhere the formula cannot see them: capital expenditure, utilisation, electricity, latency, and the operational cost of a model you now maintain.

Two consequences worth carrying over:

• The cascade case for local is stronger than for cheap APIs, not weaker. If you already own the hardware, the escalation tolerance is essentially total.
• The cache trap gets worse, not better. A hosted model with a warm cached prefix is competing against $0.50/M, and reaching for local means re-processing the whole prefix somewhere with no cache at all. Which is why the continuity complaint in that thread — the incumbent model losing track of what happened — is the same problem in a different costume.

Privacy, data residency and offline operation are legitimate reasons to route locally. Marginal cost per token is not a reason, it is a category error.

What to actually do

A procedure, ordered by how much of the benefit it captures per unit of work, and written to survive a repricing.

1. Compare on effective rates, never list prices. Blend your own input/output split and cache hit rate before ranking anything. On a 92/8 agentic split the ranking is not the ranking you get from the input column, and the tokenizer skew above means it is not even the ranking you get from the output column. Our LLM API calculator does this blend across every live-priced model.

2. Cut along workload boundaries first. Classification, extraction, formatting, summarisation, routing decisions themselves, bulk offline processing: assign these statically to the cheap tier and never revisit. This is unglamorous, free to run, and captures most of the available saving. It is also literally Anthropic’s published cost guidance — Haiku for simple tasks, Sonnet for most production workloads, Opus for the hardest reasoning — and it is what LLMRouterBench’s “careful model curation beats bigger ensembles” finding recommends.

3. Preserve cache affinity above everything. Pick the model at the start of a session and hold it. If a session must change tier, change it at a natural boundary where the prefix is being rebuilt anyway. Do not let a router make per-turn decisions inside a warm conversation. If you use a gateway, turn on session pinning.

4. Build the failure detector before the classifier. You have 60-99% escalation headroom depending on the pair. Spend the engineering there: schema validation, unit tests on generated code, groundedness checks against retrieved sources, confidence thresholds, review queues. A cascade with a good detector and a dumb router beats a clever router with no detector, on both cost and quality.

5. Measure f on replay, not in production. Sample real requests, run them through the candidate cheap model offline, grade against a task-specific check. That gives you the actual failure rate to test against your tolerance. Offline means it costs no latency and can be re-run cheaply.

6. Re-run the thresholds after every price change. Which in 2026 means roughly monthly.

What will break this advice

Stated plainly, because everything above has a shelf life.

Prices move faster than architecture. GPT-5.6 Luna dropped 80% in a single announcement and changed its family’s internal spread by 5x. Any routing threshold is a function of a price ratio, and price ratios in this market have a half-life of weeks. Encode the ratio, not the decision — a threshold that reads from a rate table survives a repricing; one with a model name hard-coded in a conditional does not.

Vendors are absorbing routing. The in-model effort dial already replaced external routing for close-together tiers. If Anthropic or OpenAI ships genuine per-request adaptive tier selection inside their own API — with the cache continuity a third party structurally cannot offer — the third-party predictive router loses its remaining case. One commenter in the June thread made exactly this prediction, and it is the correct thing to worry about if you are building one.

The cache mechanics are vendor policy, not physics. The 10x read discount, the 1.25x write, the five-minute TTL: all of them are pricing decisions that have already changed once this year. OpenAI began charging for cache writes on the GPT-5.6 family after years of not charging at all. If write rates fall, the cost of switching models falls with them and the case for fine-grained routing improves.

What will not break: the escalation formula, because it is arithmetic; the observation that a cached token on an expensive model can undercut a fresh token on a cheap one, because it follows from any discount deeper than the tier gap; and the rule that detection, not classification, is the constraint. Those hold whatever the rates do.

The single-sentence version: routing is not a cost problem wearing a machine-learning hat. It is a quality-assurance problem with a cost side-effect, and the money leaks through the cache, not the classifier.

Rates in this post come from our pricing snapshot, verified against vendor pricing pages on August 2-3, 2026, and all blended figures are computed with the same engine that powers the calculator. Research citations are dated at first mention. If a number here has drifted, tell us — corrections are logged publicly.