For the first release of AI Cost Bench, on 14 November 2026, we sent the same 82 work tasks to ten AI models, each through its maker’s own API, and let a program check every answer. Three of the ten listed exactly the same price: $0.10 per million input tokens and $0.50 per million output. Their bills for the whole basket came to $0.079 on GPT-6 Luna, $0.111 on Claude Haiku 5.5 and $0.303 on Muse Ember. The cheapest of the three got 10 tasks wrong. The most expensive got none wrong.
A price sheet tells you what a token costs. It does not tell you how many tokens a model will spend on your task, how much of its input will come back from cache, how much of its output is reasoning nobody reads, or how often the answer is wrong. Those four numbers decide what a finished task costs. Below is how far apart each one was on identical work, and how to measure them on yours.
Every figure here comes from AI Cost Bench, our own benchmark: 82 tasks of six kinds (support triage, invoice extraction, answers from long documentation, a back-office agent with tools, SQL analysis, bug fixing), one run per model, billed at each vendor’s list price on the run day. The rank correlations, the estimates and the failure-price tables are our arithmetic on the published data.
The price sheet gets the order right and the size wrong
Put the ten models of the first release in order of list price, then in order of bill, and the two orders mostly agree: the rank correlation is 0.91, with list prices blended three parts input to one part output. If all you need is a rough sort into cheap, middle and expensive, the price sheet does that.
The sizes are what it gets wrong. Blended list prices across the ten run from $0.20 to $4.00 per million tokens, 20x. Bills for the same work ran from $0.079 to $4.09, 52x. Inside the cheapest price tier, where three models listed the same input and output price, the bills were 3.8x apart.
Many cost-comparison pages go one step further: they multiply each list price by one assumed token count, the same for every model, usually with no cache. We did exactly that with the median of our own runs, about 21,500 input and 1,800 output tokens per task, and set the estimate against the measured bill:
| Model | List price per 1M, in / out | Same-token estimate | Measured bill |
|---|---|---|---|
| GPT-6.1 Terra | $1.00 / $5.00 | $2.50 | $0.537 |
| Claude Sonnet 5.5 | $2.00 / $10.00 | $4.99 | $1.56 |
| Gemini 4 Hydrogen | $1.50 / $7.50 | $3.74 | $4.09 |
| GPT-6 Luna | $0.10 / $0.50 | $0.250 | $0.079 |
| Claude Haiku 5.5 | $0.10 / $0.50 | $0.250 | $0.111 |
| Muse Ember | $0.10 / $0.50 | $0.250 | $0.303 |
The estimate was 4.7x too high for GPT-6.1 Terra and 18% too low for Muse Ember. It also put Claude Sonnet 5.5 above Gemini 4 Hydrogen, and the bills came out the other way round: Hydrogen listed 25% below Sonnet per token and billed 2.6x more. An error that size is not a rounding problem. The token count is the half of the bill the price sheet leaves out, and assuming it away throws out the part that varies most.
Same tasks, up to 5.8x the tokens
On identical prompts:
- Output per task ran from 590 tokens on GPT-6.1 Terra to 3,445 on Muse Ember, 5.8x apart.
- Input per task ran from about 15,700 tokens on Terra to 37,400 on MiniMax M3.1 Flash, 2.4x, and two things moved it. The first is steps: in the tool loops every step sends the conversation again, and the runs needed between 153 and 281 requests for the same 82 tasks. The second is counting. On the 40 support tasks, one request each with identical text, Claude Haiku 5.5 and Claude Sonnet 5.5 were billed for 481,710 input tokens and GPT-6 Luna for 329,852: 46% more tokens for the same words. Every vendor counts tokens with its own tokenizer, and you pay per token as the vendor counts it.
Muse Ember and GPT-6 Luna listed the same input and output price, and Muse’s cached rate was even lower, $0.004 against $0.01. Against Luna, Muse used 1.8x the input tokens, 3.5x the output tokens and 6.3x the reasoning tokens, and read 40% of its input from cache to Luna’s 85%. Those token counts are the whole 3.8x, and the price columns show none of them.
Tokens per task is not a number you can look up. It depends on the model, on its default settings and on your task, so the only way to know it is to measure it. Two lines on the bill moved it most: cache and reasoning.
How much of the input came back from cache
Input read from cache bills at a fraction of the input price, from 4% to about 20% across these ten models. For anything with a long shared prompt or a tool loop, the share of input that hits the cache decides more of the bill than the input price does. We made that case with arithmetic in The list price is not the price your agent pays. Here it is measured.
Four of the ten read 85% of their input from cache: GPT-6 Luna, GPT-6.1 Terra, Claude Haiku 5.5 and Claude Sonnet 5.5. Muse Ember read 40%, Gemini 4 Hydrogen 23%.
For Hydrogen that was half the bill: $2.04 of its $4.09 was input billed at the full price. As arithmetic only, had 85% of its input come from cache like the four above, the same tokens would have cost about $2.53. That is still 1.6x Sonnet’s bill, so cache alone does not explain the gap. The other half is below.
We turned caching on the way each vendor documents it: automatic where it is automatic, with cache markers where the vendor requires them, as Anthropic does. One run cannot tell us whether a cache that did not hit is a property of the model’s API or of the route the requests took, and the benchmark’s limits say so. You do not need to know which it is to fix it. Read the cached-token count from every response on your own route. If it sits near zero on a prompt that repeats, the cache is not working, and that is the cheapest fix on this list.
Reasoning you pay for and never read
Reasoning tokens bill at the output rate and never reach your user. Their share of output tokens ran from 0% on Gemini 3.5 Flash-Lite, which ran at minimal effort, to 79% on Gemini 4 Hydrogen. Qwen3.7 Plus was at 76%, Muse Ember 75%, GLM-5.3-FlashX 74%; Claude Sonnet 5.5 at 25% and GPT-6.1 Terra at 19%.
Hydrogen’s reasoning came to $1.60, more than Claude Sonnet 5.5’s whole bill of $1.56.
We set no reasoning level, so every model ran at its maker’s default, and the defaults are not alike. Muse Ember, GLM-5.3-FlashX and MiniMax M3.1 Flash default to max, Claude Sonnet 5.5 to high, four others to medium. Six of the ten could not switch reasoning off, by the model catalog of the route on the run day. Five of those six accept a lower level than the one they defaulted to; Gemini 3.5 Flash-Lite was already at its lowest.
We did not test the lower levels, so we cannot tell you what they do to the pass rate, and a lower level that fails more tasks may not save anything once failures are priced. That test is yours to run. A default of max is a choice the vendor made for you, and reasoning effort is the cost dial most teams never touch.
A wrong answer has a price, and it picks the winner
The bill is not the cost of the work. A task that comes back wrong has to be noticed and done again, and someone pays for that. Paying more did not reliably buy more right answers: the rank correlation between bill and tasks passed was 0.40, and the two runs that passed all 82 tasks cost $0.303 (Muse Ember) and $4.09 (Gemini 4 Hydrogen).
So price it. The cost of a clean result is the bill plus a fixed charge for every failed task. Who comes first depends on that charge:
| One failed task costs | 1st | 2nd | 3rd |
|---|---|---|---|
| $0 | Luna, $0.079 | Haiku 5.5, $0.111 | Muse Ember, $0.303 |
| 1 cent | Haiku 5.5, $0.141 | Luna, $0.179 | Muse Ember, $0.303 |
| 10 cents | Muse Ember, $0.303 | Haiku 5.5, $0.411 | Terra, $0.637 |
| 33 cents | Muse Ember, $0.303 | Terra, $0.870 | Haiku 5.5, $1.11 |
| $1 | Muse Ember, $0.303 | Terra, $1.54 | Haiku 5.5, $3.11 |
Luna is GPT-6 Luna, Terra is GPT-6.1 Terra, Haiku 5.5 is Claude Haiku 5.5. Each figure is the bill plus failed tasks times the charge.
The crossovers are exact. GPT-6 Luna was cheapest while a failure cost less than 0.46 cents. Claude Haiku 5.5 was cheapest from there to 6.39 cents. Above that it was Muse Ember, which failed nothing, so its clean result never moves.
Two ways to put a number on it:
- A program catches every failure and retries. Then a failure costs roughly one more attempt: about 0.1 cents on GPT-6 Luna, whose whole basket averaged $0.001 a task. Below the 0.46-cent line, the cheapest bill wins. This holds only if your checker really does catch every wrong answer, and a retry can fail too.
- A person has to notice. Thirty seconds of a reviewer’s time at $40 an hour is 33 cents a failure. At that price GPT-6 Luna, the cheapest bill in the run, ranks eighth of ten.
The benchmark page has a switch that sets the failure price anywhere from $0 to $1 and re-sorts the table. Its default is 10 cents.
The winner changes with the kind of work
At 10 cents a failure, the cheapest clean result for each kind of task:
| Kind of work | Tasks | Cheapest clean result |
|---|---|---|
| Support queue triage | 40 | GPT-6 Luna, 40/40, $0.008 |
| Answer from documentation | 12 | Claude Haiku 5.5, 12/12, $0.016 |
| Data analyst (SQL) | 8 | Claude Haiku 5.5, 8/8, $0.018 |
| Invoice to JSON | 8 | Muse Ember, 8/8, $0.020 |
| Back-office agent with tools | 8 | Muse Ember, 8/8, $0.028 |
| Fix the bug | 6 | Muse Ember, 6/6, $0.096 |
The cheapest bill wins exactly where it does not fail. GPT-6 Luna passed all 40 support tasks and nothing beat it there. On invoices it passed 5 of 8 and dropped behind two models whose bills were less than half a cent higher.
Two cautions. Each kind is 6 to 40 tasks from a single run, too few to route production traffic by. And the spread inside one kind is wider than across the basket: on support triage the most expensive run cost 105x the cheapest. If you split traffic by kind of task, the escalation math for routing applies, and each kind needs its own measurement on your tasks.
How to run this on your own work
None of this needs our tasks. It needs yours and an afternoon.
- Collect 30 to 100 tasks from your own traffic with answers a program can check: tests that pass, JSON fields that match, a SQL result, a label. Do not grade with another model; a judge adds its own errors and its own bill.
- Send every model the same prompts in the same order. Record the reasoning level each one ran at. Run the default first, then the level you would actually ship.
- Turn caching on the way each vendor documents it, with markers where the vendor requires them. A test without caching overstates the bill of every vendor that only caches on request.
- Start every run with a cold cache. Put a random line at the very top of the system prompt, a new one for each run. When we skipped this during calibration, a rerun of GPT-6 Luna’s invoice tasks came out about a third cheaper, because it read a cache the previous run had paid to write.
- Price the bill from the usage object, not from the price sheet: input at full price, cached input, cache writes, output, reasoning. The same field name means different things at different vendors, so read each field the way its vendor defines it.
- Check the price thresholds against your prompt sizes. Claude Haiku 5.5 bills five times its base rates once a prompt passes 100,000 tokens, Muse Ember three times past 200,000. Our largest prompt was the 35,000 to 45,000-token documentation bundle, under both lines. A workload with longer prompts would rank differently.
- Put a price on a failure in your own terms, and compute the clean result.
- Treat small gaps as ties. While calibrating we ran two models six times each: one passed between 77 and 81 of the 82 tasks, the other between 72 and 80, while their bills moved by a few percent. One run is one sample. Run twice before you move a production workload.
Steps 7 and 8 are a few lines of code. This script reads our published results, which are free to reuse under CC BY 4.0, and prints who has the cheapest clean result at every failure price, then the winner for each kind of work. To use it on your own runs, give winners() a list of (name, bill, failed_tasks) from the same task set.
import json, urllib.request
URL = "https://aicost.tools/benchmarks/ai-cost-bench/data.json" # CC BY 4.0, credit AI//COST
def load(url=URL):
req = urllib.request.Request(url, headers={"User-Agent": "cost-check/1.0"})
with urllib.request.urlopen(req) as r:
return json.load(r)["models"]
def clean_cost(bill, failed, fail_usd):
"""The bill plus what it costs you to catch and redo each wrong answer."""
return bill + fail_usd * failed
def winners(runs, up_to=5.0):
"""Who has the cheapest clean result at every failure price from $0 to up_to.
runs: [(name, bill_usd, failed_tasks)] on the same task set.
Returns [(name, from_usd, to_usd)], to_usd None for the last span."""
f = 0.0
best = min(runs, key=lambda r: (r[1], r[2]))
spans = []
while True:
nxt = None
for r in runs:
if r[2] < best[2]: # only a model that fails less can overtake
x = (r[1] - best[1]) / (best[2] - r[2]) # failure price where it catches up
if x >= f and (nxt is None or (x, r[2]) < (nxt[0], nxt[1][2])):
nxt = (x, r)
if nxt is None or nxt[0] > up_to:
spans.append((best[0], f, None))
return spans
spans.append((best[0], f, nxt[0]))
f, best = nxt
models = load()
runs = [(m["model"], m["billUsd"], m["tasks"] - m["tasksPassed"]) for m in models]
for name, lo, hi in winners(runs):
end = f"to {hi * 100:.2f} cents" if hi is not None else "and up"
print(f"{name:20} cheapest from {lo * 100:.2f} cents per failure {end}")
FAIL_USD = 0.10 # set your own: minutes to catch and redo a wrong answer x hourly cost
for kind in models[0]["byTaskType"]:
cells = [(m["model"], m["byTaskType"][kind]["billUsd"],
m["byTaskType"][kind]["tasks"] - m["byTaskType"][kind]["passed"]) for m in models]
name, bill, failed = min(cells, key=lambda c: clean_cost(c[1], c[2], FAIL_USD))
print(f"{kind:16} {name:20} ${clean_cost(bill, failed, FAIL_USD):.3f} clean, {failed} failed")
Run on 18 November 2026, against the ten models of the first release, its first three lines read. With more models in the table, the names and the break points will be different; the way to read them will not:
GPT-6 Luna cheapest from 0.00 cents per failure to 0.46 cents
Claude Haiku 5.5 cheapest from 0.46 cents per failure to 6.39 cents
Muse Ember cheapest from 6.39 cents per failure and up
winners() walks the lower edge of the lines “bill plus price times failures”. At each step the next leader is the model that fails less and catches up at the lowest failure price. A model that is never cheapest at any price, however much you charge for a failure, never appears. We checked it against a brute-force scan of 50,000 failure prices between $0 and $5 on the published data.
What these numbers do not tell you
- A first release. Ten models, runs of 14 November 2026. Models added later, and later runs of these ten, appear on the benchmark page, not in this text.
- One run per model. Pass counts move by a few tasks between runs, which is why a gap of 4 tasks or less is a tie and why we price failures at 10 cents rather than a dollar on the benchmark page.
- Default settings. A team that tunes the reasoning level or the prompt for one model can get a lower bill or a better result than its default gave us.
- List prices only. No batch, flex or priority tiers, no committed-use discounts. Qwen3.7 Plus was actually charged 20% less than its list-price bill because of a limited-time discount; we count the list price.
- Synthetic tasks. Invented companies, generated invoices and databases, written so a program knows the right answer. Audio, video and agents that run for hours are not covered.
- One route. Requests went through OpenRouter, pinned to each maker’s own endpoint with fallbacks off.
The method is on the benchmark page, and our methodology explains how every list price on this site is read and dated.
The price sheet is a fine first filter: it sorts models into cheap, middle and expensive about right. It cannot pick between the models that survive the filter, because what separates them is how many tokens they spend, how much of their input comes back from cache, how much they reason, and how often they are wrong. Those come from running your own tasks. The leaders on the benchmark page will change as models are added; those four questions will not.
| Model | Input /M | Cached input /M | Output /M |
|---|---|---|---|
| GPT-6 Luna OpenAI | $0.1 | $0.01 | $0.5 |
| Claude Haiku 5.5 Anthropic | $0.1 | $0.01 | $0.5 |
| Muse Ember Meta | $0.1 | $0.004 | $0.5 |
| GPT-6.1 Terra OpenAI | $1 | $0.05 | $5 |
| Claude Sonnet 5.5 Anthropic | $2 | $0.1 | $10 |
| Gemini 4 Hydrogen Google | $1.5 | $0.075 | $7.5 |
Sources: AI Cost Bench, release 2026-11, runs of 14 November 2026, with the published data.json (CC BY 4.0). List prices read on the vendors’ pages on 18 November 2026: OpenAI, Anthropic, Meta and Google. The rank correlations, the same-token estimate and the failure-price tables are our own arithmetic on the published data. See corrections for what we have got wrong.
Frequently asked questions
What is cost per task for an LLM?
What one finished piece of work costs: every token the model spent on it, at that model's prices, including cached input, cache writes and reasoning. It differs from the price per million tokens because models spend very different numbers of tokens on the same work. In the first release of AI Cost Bench, output per task ran from 590 tokens on GPT-6.1 Terra to 3,445 on Muse Ember. The useful version also counts failures: the cost of a clean result is the bill plus a charge for every task that came back wrong.
Why can a model with a lower price per token cost more?
Because the bill is price times tokens, and the price sheet only shows the price. In the first release of AI Cost Bench, Gemini 4 Hydrogen listed at $1.50 input and $7.50 output per million, 25% below Claude Sonnet 5.5's $2 and $10. On the same 82 tasks Hydrogen billed $4.09 and Sonnet $1.56. Two lines did it: 79% of Hydrogen's output tokens were reasoning, and only 23% of its input came from cache, against 85% for Sonnet. Half of Hydrogen's bill, $2.04, was input at the full price.
How do I estimate what a model will cost on my workload?
Run it. Take 30 to 100 tasks from your own traffic whose answers a program can check, send the same prompts to every model you are considering, turn caching on as each vendor documents it, start each run with a cold cache, and price the bill from the token counts in each response. A spreadsheet that multiplies list prices by one assumed token count will not get you there: on our runs that method was off by up to 4.7x and ranked two models the wrong way round.
How much should I count for a failed task?
What it costs you to notice the failure and get the task done again. If a program checks every answer and retries automatically, a failure costs about one more attempt, a fraction of a cent on a small model. If a person has to catch it, the price is their time: thirty seconds at $40 an hour is 33 cents. In the first release of AI Cost Bench the cheapest model changed at 0.46 cents and again at 6.39 cents per failure, so the number you pick decides the ranking.
How many test tasks do I need to compare two models?
Enough that the gap you care about is bigger than the run-to-run wobble. When we ran two models six times each on the same 82 tasks, one passed between 77 and 81 and the other between 72 and 80, while their bills moved by only a few percent. Bills repeat closely; pass counts do not. Treat a gap of about 5% of the tasks or less as a tie, and run twice before you move a production workload.
Does reasoning effort change the bill?
Yes, and the default is not yours to assume. We set no reasoning level, so each model ran at its maker's default. In the first release three defaulted to max, one to high, four to medium and one to minimal, and one has no named levels. Reasoning tokens bill at the output rate and made up anywhere from 0% to 79% of the output. Six of the ten could not switch reasoning off on the run day's route, but five of those six accept a lower level than the one they defaulted to. Test the level you would ship, not only the default.
Will these numbers change as AI Cost Bench adds models?
The benchmark will; this article will not. AI Cost Bench started in November 2026 with ten models and grows as we run more, and each model's row on the benchmark page shows its latest run, so the ranking there moves. The figures in this article are the first release, runs of 14 November 2026, kept as measured because they illustrate the method. To see where the break points sit today, run the script in this article: it reads the live results.
Can I use the AI Cost Bench numbers?
Yes. The results, charts and text are licensed CC BY 4.0: quote, re-chart and republish them, commercially too, with credit to AI Cost Bench by AI//COST and a link to the benchmark page. The full results are published as JSON and CSV, and the script in this article reads the JSON directly.
Nothing yet. Mention this post on any platform — Mastodon, Bluesky, LinkedIn, a blog — and the citation surfaces here.