Last verified
Original benchmarkBy AI//COSTOne run per modelCC BY 4.0

AI Cost Bench

As of November 21, 2026, the same 82 work tasks cost $0.080 on GPT-6 Luna and $4.09 on Gemini 4 Hydrogen. Each model gets identical tasks through its maker's own API, a program checks every answer, and the bill is the tokens the vendor reported at its list price. With a failed task priced at 10 cents, Hunyuan Hy4 Flash gets the work done cheapest: $0.239 for a clean result.

Top of the index
343tasks / $
Hunyuan Hy4 Flash · 81/82 passed
Cheapest bill
$0.080
GPT-6 Luna · 72/82 passed
Most expensive bill
$4.09
Gemini 4 Hydrogen · 82/82 passed
Spread
51x
same tasks, same rules
Tasks
82
6 kinds of work
Models
17
12 vendors

What is AI Cost Bench? Our own test of what everyday work costs on each AI model. Most benchmarks ask how hard a problem a model can solve; this one prices the ordinary jobs: the support letters, documentation questions, invoices, back-office cases, SQL reports and bug fixes a product hands to a model over and over. Price lists say what a million tokens cost; this says what a finished job costs, because every model runs the same 82 tasks and we add up what it was billed. The ranking adds 10 cents for every task a model got wrong, so a cheap model that fails does not win. It is for teams that run AI inside a product or a workflow and pay per token, where one kind of task comes in thousands of times and what matters is the cost of doing it right.

What the November 2026 runs show

§ 01 / INDEX

AI Cost Bench Index

Work tasks done right per dollar · November 2026 · one run per model

Embed Table
  1. 343 Hunyuan Hy4 Flash
  2. 337 MiMo-V2.6-Flash
  3. 271 Muse Ember (max)
  4. 263 Claude Haiku 5.5 (medium)
  5. 179 GLM-5.3-Flash (max)
  6. 176 DeepSeek V4.1 Flash (high)
  7. 127 GPT-6.1 Terra (medium)
  8. 102 GLM-5.3-FlashX (max)
  9. 90 MiniMax M3.1 Flash (max)
  10. 76 GPT-6 Luna (medium)
  11. 68 Qwen3.7 Plus
  12. 47 Step 5 Preview (medium)
  13. 46 Claude Sonnet 5.5 (high)
  14. 38 Kimi K3.1 Pro (max)
  15. 29 Muse Spark 1.3 (medium)
  16. 20 Gemini 4 Hydrogen (medium)
  17. 18 Gemini 3.5 Flash-Lite (minimal)
Higher is better. The index is how many of the 82 tasks a dollar gets done right: 82 ÷ (the bill + 10 cents for every failed task). Hunyuan Hy4 Flash leads at 343; MiMo-V2.6-Flash, Muse Ember, Claude Haiku 5.5, GLM-5.3-Flash and DeepSeek V4.1 Flash are inside the spread between repeat runs of one model, so read them as level with it. The word in brackets is the reasoning effort of the run: the model's default, we set none. Free to repost: the PNG and the embed code carry the credit.
§ 02 / RESULTS

What the basket cost, model by model

17 models · highest index first

The index is built on the cost of a clean result: the bill plus 10 cents for every task the model got wrong. A failed task is not free, someone has to catch it and do it again, and without that charge the cheapest bill wins even when a tenth of the work comes back wrong. Divide the 82 tasks by that cost and you get the index: tasks done right per dollar. 10 cents is our assumption; set your own price and the table re-sorts.

A failed task costs
# Model Index ↓ Clean result Tasks passed Bill for the basket List price / 1M Reasoning From cache Run
1 Hunyuan Hy4 Flash Tencent (Hunyuan) 343 $0.239 bill + 1 × 10¢
81/82
$0.139
$0.07 / $0.281 78% 42% 24 min · 215 requests
2 MiMo-V2.6-Flash Xiaomi MiMo 337 $0.244 bill + 1 × 10¢
81/82
$0.144
$0.14 / $0.28 78% 74% 71 min · 209 requests
3 Muse Ember Meta · max effort 271 $0.303 nothing failed
82/82
$0.303
$0.1 / $0.5 75% 40% 13 min · 273 requests
4 Claude Haiku 5.5 Anthropic · medium effort 263 $0.312 bill + 2 × 10¢
80/82
$0.112
$0.1 / $0.5 47% 84% 10 min · 163 requests
5 GLM-5.3-Flash Zhipu (Z.ai / GLM) · max effort 179 $0.457 bill + 2 × 10¢
80/82
$0.257
$0.15 / $0.5 80% 50% 42 min · 173 requests
6 DeepSeek V4.1 Flash DeepSeek · high effort 176 $0.465 bill + 2 × 10¢
80/82
$0.265
$0.3 / $1.2 69% 90% 15 min · 196 requests
7 GPT-6.1 Terra OpenAI · medium effort 127 $0.646 bill + 1 × 10¢
81/82
$0.546
$1 / $5 19% 84% 13 min · 172 requests
8 GLM-5.3-FlashX Zhipu (Z.ai / GLM) · max effort 102 $0.807 bill + 4 × 10¢
78/82
$0.407
$0.37 / $1.25 74% 78% 23 min · 167 requests
9 MiniMax M3.1 Flash MiniMax · max effort 90 $0.907 bill + 6 × 10¢
76/82
$0.307
$0.15 / $0.6 62% 75% 27 min · 281 requests
10 GPT-6 Luna OpenAI · medium effort 76 $1.08 bill + 10 × 10¢
72/82
$0.080
$0.1 / $0.5 42% 85% 16 min · 194 requests
11 Qwen3.7 Plus Alibaba (Qwen) 68 $1.21 bill + 5 × 10¢
77/82
$0.705
$0.4 / $1.6 76% 81% 73 min · 251 requests
12 Step 5 Preview StepFun · medium effort 47 $1.75 bill + 6 × 10¢
76/82
$1.15
$1 / $2.7 not reported 76% 32 min · 216 requests
13 Claude Sonnet 5.5 Anthropic · high effort 46 $1.79 bill + 2 × 10¢
80/82
$1.59
$2 / $10 25% 84% 10 min · 153 requests
14 Kimi K3.1 Pro Moonshot (Kimi) · max effort 38 $2.17 nothing failed
82/82
$2.17
$2 / $10 67% 82% 47 min · 179 requests
15 Muse Spark 1.3 Meta · medium effort 29 $2.80 bill + 2 × 10¢
80/82
$2.60
$1.25 / $4.25 68% 45% 27 min · 291 requests
16 Gemini 4 Hydrogen Google DeepMind · medium effort 20 $4.09 nothing failed
82/82
$4.09
$1.5 / $7.5 79% 23% 24 min · 211 requests
17 Gemini 3.5 Flash-Lite Google · minimal effort 18 $4.51 bill + 42 × 10¢
40/82
$0.313
$0.3 / $2.5 0% 73% 5 min · 243 requests

The bill is every token the vendor reported for the 82 tasks at its list price today, input and output, cache reads and writes included. A task counts as passed only when every part of the answer is right. The clean result adds 10 cents for every failed task. At that price a gap under $0.40 is 4 tasks, which is inside the spread between repeat runs of one model: read MiMo-V2.6-Flash, Muse Ember, Claude Haiku 5.5, GLM-5.3-Flash and DeepSeek V4.1 Flash as level with Hunyuan Hy4 Flash. Below 0.4 cents per failed task GPT-6 Luna comes first, from 0.4 cents to 2.7 cents Claude Haiku 5.5, from 2.7 cents to 16.4 cents Hunyuan Hy4 Flash, above 16.4 cents Muse Ember. List prices come from our price index; the oldest of them was read on the vendor's page on 2026-10-21.

§ 03 / BILL VS RESULT

The bill against the result

  • Pareto line: no cheaper model does better
  • Cheaper and better than the median
40% 60% 80% 100% $0.100 $0.300 $1.00 $3.00 Bill for the 82 tasks, USD (log scale) Tasks passed Hunyuan Hy4 Flash, Tencent (Hunyuan): 98.8% at $0.139 MiMo-V2.6-Flash, Xiaomi MiMo: 98.8% at $0.144 Muse Ember (max), Meta: 100% at $0.303 Claude Haiku 5.5 (medium), Anthropic: 97.6% at $0.112 GLM-5.3-Flash (max), Zhipu (Z.ai / GLM): 97.6% at $0.257 DeepSeek V4.1 Flash (high), DeepSeek: 97.6% at $0.265 GPT-6.1 Terra (medium), OpenAI: 98.8% at $0.546 GLM-5.3-FlashX (max), Zhipu (Z.ai / GLM): 95.1% at $0.407 MiniMax M3.1 Flash (max), MiniMax: 92.7% at $0.307 GPT-6 Luna (medium), OpenAI: 87.8% at $0.080 Qwen3.7 Plus, Alibaba (Qwen): 93.9% at $0.705 Step 5 Preview (medium), StepFun: 92.7% at $1.15 Claude Sonnet 5.5 (high), Anthropic: 97.6% at $1.59 Kimi K3.1 Pro (max), Moonshot (Kimi): 100% at $2.17 Muse Spark 1.3 (medium), Meta: 97.6% at $2.60 Gemini 4 Hydrogen (medium), Google DeepMind: 100% at $4.09 Gemini 3.5 Flash-Lite (minimal), Google: 48.8% at $0.313 Hunyuan Hy4 FlashMiMo-V2.6-FlashMuse Ember (max)Claude Haiku 5.5 (medium)GLM-5.3-Flash (max)DeepSeek V4.1 Flash (high)GPT-6.1 Terra (medium)GLM-5.3-FlashX (max)MiniMax M3.1 Flash (max)GPT-6 Luna (medium)Qwen3.7 PlusStep 5 Preview (medium)Claude Sonnet 5.5 (high)Kimi K3.1 Pro (max)Muse Spark 1.3 (medium)Gemini 4 Hydrogen (medium)Gemini 3.5 Flash-Lite (minimal)
The dashed Pareto line joins the models no cheaper model matched, whoever made them: Hunyuan Hy4 Flash (81 of 82, $0.139), Muse Ember (82 of 82, $0.303), Claude Haiku 5.5 (80 of 82, $0.112) and GPT-6 Luna (72 of 82, $0.080). A model below the line cost more than another that passed at least as many tasks. The mark on a point is the model's maker, the word in brackets the reasoning effort of the run. The vertical axis is the share of the 82 tasks passed; a gap of 4 tasks (5 points) or less is within the movement we saw between repeat runs of one model.
§ 04 / WHERE THE MONEY WENT

Where the money went

A price list has two numbers per model. A bill has five lines. Reasoning tokens are billed as output and never shown; input read from cache costs a fraction of the input price, when the cache hits. On these tasks GLM-5.3-Flash spent 80% of its output on reasoning and Gemini 3.5 Flash-Lite 0%; DeepSeek V4.1 Flash read 90% of its input from cache and Gemini 4 Hydrogen 23%.

Model Share of the bill Input at full priceInput from cacheCache writesAnswer tokensReasoning tokens Bill
Hunyuan Hy4 Flash $0.072$0.0022—$0.014$0.050 $0.139
MiMo-V2.6-Flash $0.072$0.0041—$0.015$0.052 $0.144
Muse Ember $0.157$0.0042—$0.035$0.107 $0.303
Claude Haiku 5.5 $0.0067$0.016$0.029$0.033$0.029 $0.112
GLM-5.3-Flash $0.117$0.023—$0.024$0.093 $0.257
DeepSeek V4.1 Flash $0.048$0.0085—$0.064$0.145 $0.265
GPT-6.1 Terra $0.011$0.054$0.239$0.196$0.045 $0.546
GLM-5.3-FlashX $0.112$0.082—$0.056$0.157 $0.407
MiniMax M3.1 Flash $0.115$0.035—$0.060$0.098 $0.307
GPT-6 Luna $0.0014$0.012$0.026$0.023$0.017 $0.080
Qwen3.7 Plus $0.168$0.140—$0.095$0.301 $0.705
Step 5 Preview $0.440$0.136—$0.578 all output, split not reported $1.15
Claude Sonnet 5.5 $0.135$0.148$0.516$0.592$0.198 $1.59
Kimi K3.1 Pro $0.525$0.235—$0.464$0.944 $2.17
Muse Spark 1.3 $1.60$0.155—$0.266$0.575 $2.60
Gemini 4 Hydrogen $2.04$0.031—$0.417$1.60 $4.09
Gemini 3.5 Flash-Lite $0.132$0.042—$0.139— $0.313
§ 05 / BY KIND OF WORK

Result and bill by kind of work

Tasks passed on each of the 6 kinds of work, and what the work costs. The kinds differ in size, from 6 to 40 tasks, so the bill is shown for 1,000 tasks of a kind, scaled up from these runs; the switch shows the bill as measured. A thousand support letters cost about $0.23 on GPT-6 Luna and $22 on Gemini 4 Hydrogen, 94x apart, and both passed all 40 here. A model that is cheap on one kind can be the expensive one on another: loops with tools multiply every token by the number of steps.

Bill shown
Model Bug fix 6Invoices 8Support 40Docs Q&A 12Agent 8SQL 8
Hunyuan Hy4 Flash 6/6 $5.3 8/8 $2.1 40/40 $0.83 12/12 $3.2 8/8 $1.1 7/8 $1.3
MiMo-V2.6-Flash 5/6 $11 8/8 $2.0 40/40 $0.55 12/12 $1.3 8/8 $1.5 8/8 $1.8
Muse Ember 6/6 $16 8/8 $2.5 40/40 $1.4 12/12 $4.4 8/8 $3.4 8/8 $6.1
Claude Haiku 5.5 5/6 $3.7 8/8 $2.6 40/40 $0.49 12/12 $1.4 7/8 $1.9 8/8 $2.3
GLM-5.3-Flash 6/6 $13 8/8 $3.6 40/40 $1.7 11/12 $1.9 8/8 $3.6 7/8 $4.0
DeepSeek V4.1 Flash 6/6 $15 7/8 $6.1 40/40 $1.1 11/12 $1.8 8/8 $4.2 8/8 $3.6
GPT-6.1 Terra 6/6 $29 7/8 $16 40/40 $1.6 12/12 $6.3 8/8 $6.7 8/8 $6.2
GLM-5.3-FlashX 5/6 $17 7/8 $7.9 40/40 $1.9 10/12 $5.4 8/8 $6.6 8/8 $5.8
MiniMax M3.1 Flash 5/6 $22 7/8 $3.5 39/40 $0.94 11/12 $3.8 7/8 $2.9 7/8 $4.9
GPT-6 Luna 5/6 $5.1 5/8 $2.0 40/40 $0.23 9/12 $0.85 7/8 $0.99 6/8 $0.74
Qwen3.7 Plus 3/6 $27 8/8 $12 40/40 $3.2 11/12 $6.8 8/8 $12 7/8 $17
Step 5 Preview 6/6 $70 6/8 $23 38/40 $4.0 11/12 $9.5 7/8 $12 8/8 $23
Claude Sonnet 5.5 5/6 $64 8/8 $42 40/40 $5.8 12/12 $20 7/8 $30 8/8 $20
Kimi K3.1 Pro 6/6 $115 8/8 $54 40/40 $6.6 12/12 $18 8/8 $42 8/8 $28
Muse Spark 1.3 6/6 $92 8/8 $23 40/40 $13 12/12 $49 8/8 $36 6/8 $57
Gemini 4 Hydrogen 6/6 $196 8/8 $43 40/40 $22 12/12 $71 8/8 $47 8/8 $58
Gemini 3.5 Flash-Lite 0/6 $16 2/8 $6.7 29/40 $0.58 3/12 $1.7 4/8 $4.5 2/8 $10
§ 06 / TASK BY TASK

Every task, every model

One square is one task. 32 of the 82 tasks were passed by every model. The task most models missed is in answer from documentation: 5 models of 17 failed it.

Model Bug fixInvoicesSupportDocs Q&AAgentSQL Failed
Hunyuan Hy4 Flash 1
MiMo-V2.6-Flash 1
Muse Ember 0
Claude Haiku 5.5 2
GLM-5.3-Flash 2
DeepSeek V4.1 Flash 2
GPT-6.1 Terra 1
GLM-5.3-FlashX 4
MiniMax M3.1 Flash 6
GPT-6 Luna 10
Qwen3.7 Plus 5
Step 5 Preview 6
Claude Sonnet 5.5 2
Kimi K3.1 Pro 0
Muse Spark 1.3 2
Gemini 4 Hydrogen 0
Gemini 3.5 Flash-Lite 42

passed failed · hover a square for the task

§ 07 / THE TASKS

The 82 tasks, 6 kinds of work

The 82 tasks themselves are not published, on purpose. A public test set ends up in training data, and from then on a model can pass a task because it has seen the answer. A closed set stays a test of the work. What is public is everything needed to judge the method: what each kind of task asks for and how it is checked (below), the rules of a run, for every model exactly which tasks it failed (the grid above), and one sample task of each kind, written separately and shown in full.

Fix the bug

6 tasks · Tool loop, up to 25 steps

Six small JavaScript libraries of 250 to 600 lines: invoice rounding, an interval scheduler, an LRU cache with expiry, a CSV parser, a rate limiter, semver ranges. Each has two or three planted bugs and a bug report that describes symptoms, not causes. At least one bug in each project is invisible to the tests the model can run and follows only from the README.

CheckedHidden tests, 8 to 15 per project, and the visible ones must all pass in a sandbox.

Invoice to JSON

8 tasks · One request with images

A two-page invoice arrives as two page images: 18 to 30 line items, three layouts, three number formats, several currencies. The model returns every header field, every line and every total as JSON. Six of the eight invoices carry one planted arithmetic error, and the model has to name where it is and give both numbers.

CheckedEvery header field, every line, every total and the reported inconsistency must match. Numbers get a tolerance of half a cent.

Support queue triage

40 tasks · One request, long shared policy

A support policy of six to eight thousand tokens: categories, a priority matrix, refund rules with windows, exceptions and an order of precedence, escalation conditions. Each task is one customer letter of 80 to 250 words, in English, German, Spanish, French or Polish. The model returns category, priority, team, refund eligibility with the rule that decides it, order id, SKU, language and whether to escalate.

CheckedAll fields must equal the answer a rules engine computes from the letter's facts. One wrong field fails the task.

Answer from documentation

12 tasks · One request, long shared document

A documentation bundle of 35 to 45 thousand tokens for a fictional API platform: reference, a changelog of six versions with reversals, plans and quotas, SLA, a migration guide. Every question needs at least two sections: a value a later version overrode, a limit that depends on the plan, a quota times a price.

CheckedThe answer must match after normalisation and the cited sections must include the ones that decide it.

Back-office agent with tools

8 tasks · Tool loop, up to 30 steps

A shop back office with ten tools: search and read orders, check stock, refund, replace, change an address, add a note, send an email, escalate. The model gets a written policy and one customer email, and has to resolve the case with four to nine actions. The traps are the ones a real desk meets: the replacement is out of stock, the order has already shipped, the refund was issued last week, the email mentions someone else's order.

CheckedThe final state is checked: refunds to the cent, replacements, address changes, notes, escalations, the email and its outcomes, and that nothing forbidden was done.

Data analyst

8 tasks · Tool loop, up to 20 steps

Two SQLite databases, a shop and a subscription service, with the dirt real data has: test accounts, duplicates, NULLs, partial refunds, currency conversion by date, day boundaries in UTC. The model explores the schema and runs SELECTs to answer a business question with a small table of numbers.

CheckedKeys, row order and every number must match: counts exactly, money within five cents.

§ 08 / SAMPLE TASKS

One task of each kind, in full

These 6 tasks were written for this page and are not among the 82. Each went through the same checker as the real ones: its reference answer passes and a copy with one mistake fails. Open one to see what the model is given, the answer the checker wants, and where models go wrong.

Fix the bug A pagination library with three planted bugs 14 parts checked

The model sees the bug report first. Through four tools (list_files, read_file, write_file, run_tests) it can read the project, change the source and run the 5 visible tests; it has up to 25 steps. When it stops, the project is checked against those tests and 9 hidden ones written from the README.

The bug report

# The last page of results shows a wrong range

A search with 45 results, 10 per page: on page 5 the footer says "Showing 41-50 of 45" and the
counter above the list says 10 items. Pages 1 to 4 are right. Customers have started to send
screenshots.

README.md, the specification

# pagination

Page arithmetic for a list shown a page at a time, and the row of page links under it. Pages and
item numbers start at 1. Every argument is an integer; anything else throws a `RangeError`.

## API (`src/index.mjs`)

- `pageCount(total, pageSize)`
- `pageSlice(total, pageSize, page)`
- `pageWindow(current, pages, width)`
- `pageLinks(current, pages, width)`

## Behaviour

1. Page count. `pageCount(total, pageSize)` is the number of pages needed for `total` items. A list
   that fills its last page exactly has no extra page. An empty list has one page, an empty one:
   `pageCount(0, 25)` is `1`. `total` below 0 or `pageSize` below 1 throws `RangeError`.
2. Slice. `pageSlice(total, pageSize, page)` returns `{ page, from, to, count }`: the page shown, the
   numbers of its first and last item (inclusive) and how many items it holds. The last page ends at
   the last item of the list, so it can hold fewer than `pageSize` items. A `page` below 1 is treated
   as page 1 and a `page` past the end as the last page. For an empty list the result is
   `{ page: 1, from: 0, to: 0, count: 0 }` whatever `page` is.
3. Window. `pageWindow(current, pages, width)` returns the page numbers to show around the current
   page, in ascending order: `width` consecutive pages, or all of them when the list has fewer. The
   window is centred on the current page; when `width` is even, the extra page goes after the current
   one (`pageWindow(10, 20, 4)` is `[9, 10, 11, 12]`). Near the first or the last page the window is
   shifted, not cut: it always holds the same number of pages and always contains the current one.
   `current` outside `1..pages` throws `RangeError`.
4. Links. `pageLinks(current, pages, width)` is the window with the first and the last page added.
   Pages left out between an end and the window are shown as one `null`. A gap of exactly one page is
   not worth a gap mark: that page is listed instead.

## Tests

`test/visible/` holds a few examples. They do not cover every rule above.

src/pages.mjs, as shipped

// Page arithmetic for a paginated list. Item numbers are 1-based and inclusive.

export function assertInt(name, value, min) {
  if (!Number.isInteger(value) || value < min) throw new RangeError(`${name} must be an integer >= ${min}`);
}

/** Number of pages of a list. An empty list still has one page, an empty one. */
export function pageCount(total, pageSize) {
  assertInt('total', total, 0);
  assertInt('pageSize', pageSize, 1);
  return Math.ceil(total / pageSize);
}

/**
 * What a page shows: `{ page, from, to, count }`. A page number outside the list is clamped to the
 * first or the last page.
 */
export function pageSlice(total, pageSize, page) {
  const pages = pageCount(total, pageSize);
  if (!Number.isInteger(page)) throw new RangeError('page must be an integer');
  if (total === 0) return { page: 1, from: 0, to: 0, count: 0 };
  const current = Math.min(Math.max(page, 1), pages);
  const from = (current - 1) * pageSize + 1;
  const to = current * pageSize;
  return { page: current, from, to, count: to - from + 1 };
}

src/window.mjs, as shipped

// The row of page links under a list: a window of pages around the current one, plus the ends.
import { assertInt } from './pages.mjs';

/**
 * The page numbers to show around `current`: consecutive pages, as many as `width` allows and never
 * more than there are. The window is centred on the current page; with an even number of pages the
 * extra one goes after it. Near either end the window is shifted so that it keeps its size.
 */
export function pageWindow(current, pages, width) {
  assertInt('pages', pages, 1);
  assertInt('width', width, 1);
  if (!Number.isInteger(current) || current < 1 || current > pages) {
    throw new RangeError('current must be a page between 1 and pages');
  }
  const size = Math.min(width, pages);
  const start = Math.max(1, current - Math.floor((size - 1) / 2));
  return Array.from({ length: size }, (_, i) => start + i).filter((page) => page <= pages);
}

/**
 * The window with the first and the last page added. `null` stands for a gap of two or more pages
 * left out; a gap of exactly one page is filled with that page.
 */
export function pageLinks(current, pages, width) {
  const win = pageWindow(current, pages, width);
  const first = win[0];
  const last = win[win.length - 1];
  const out = [];
  if (first > 1) {
    out.push(1);
    if (first === 3) out.push(2);
    else if (first > 3) out.push(null);
  }
  out.push(...win);
  if (last < pages) {
    if (last === pages - 2) out.push(pages - 1);
    else if (last < pages - 2) out.push(null);
    out.push(pages);
  }
  return out;
}

What a complete fix changes

1. The last page ends at page x pageSize instead of at the last item, so a short last page claims a full page of items. (a visible test fails on it)
src/pages.mjs
-   const to = current * pageSize;
+   const to = Math.min(current * pageSize, total);

2. An empty list reports zero pages; the README says it has one empty page. (no visible test touches it; it follows from the README)
src/pages.mjs
-   return Math.ceil(total / pageSize);
+   return Math.max(1, Math.ceil(total / pageSize));

3. Near the last page the window is cut short instead of shifted, so it shows fewer pages than its width. (no visible test touches it; it follows from the README)
src/window.mjs
-   const start = Math.max(1, current - Math.floor((size - 1) / 2));
-   return Array.from({ length: size }, (_, i) => start + i).filter((page) => page <= pages);
+   const start = Math.max(1, Math.min(current - Math.floor((size - 1) / 2), pages - size + 1));
+   return Array.from({ length: size }, (_, i) => start + i);

What it tests. The report describes one symptom, and fixing it turns every visible test green. 2 more bugs remain, and the only way to find them is to read the README against the code: an empty list must have one page, and the window of page links must shift near the last page instead of getting shorter. The instruction says so in plain words: fix the report, then make the library behave as the README describes. A model that stops when the visible tests pass fails the task on the hidden ones.

Invoice to JSON A two-page invoice with one planted error 30 parts checked

The model gets the two page images and the instruction in one request, with no tools and no second attempt.

The instruction

The two images are page 1 and page 2 of one invoice. Extract it into one JSON object.

Return exactly this structure (all keys required, no other keys):
{
  "invoice_number": string,
  "issue_date": string in YYYY-MM-DD,
  "due_date": string in YYYY-MM-DD,
  "currency": string, the ISO 4217 code of the invoice currency (for example "EUR"),
  "seller_name": string,
  "buyer_name": string,
  "buyer_tax_id": string,
  "line_items": [
    { "line": integer, "sku": string, "qty": number, "unit_price": number, "discount_pct": number, "tax_rate_pct": number, "line_total": number }
  ],
  "net_total": number,
  "tax_total": number,
  "gross_total": number,
  "inconsistency": null or { "line": integer or null, "field": string, "printed": number, "computed": number }
}

Field rules:
- Transcribe exactly as printed. Names, the invoice number and SKUs are copied character for character, without their labels.
- "buyer_tax_id" is the buyer's tax or VAT identification number (not the seller's): the characters printed after its label (for example after "VAT reg. no.:"). For this field only, spaces, punctuation and letter case are not scored, and the label or a tax word printed after the number (such as "MWST") may be kept or left out; every letter and digit of the number itself must be right.
- Dates: "issue_date" is the invoice issue date and "due_date" the payment due date. Ignore order dates, delivery dates and other dates. Convert to YYYY-MM-DD.
- Numbers are JSON numbers: "." as the decimal separator, no thousands separators, no currency symbols, no percent signs. Read each number with the number format of the page (a space, a dot or a comma can be a thousands separator or a decimal mark depending on the locale).
- "line_items" holds every row of the item table from both pages, in printed order. "line" is the printed row number. Rows such as subtotals, "brought forward" or "carried forward" are not items.
- "qty" is the quantity, "unit_price" the price per unit before any discount, "line_total" the amount printed in the last column of the row.
- "discount_pct" is the line discount in percent; it is 0 when the row shows no discount. "tax_rate_pct" is the line tax rate in percent; it is 0 for zero-rated or exempt rows.
- "net_total" is the invoice total before tax, "tax_total" the total tax, "gross_total" the final total payable including tax (not an early-payment amount).

Arithmetic check. Test exactly these five things:
  (a) for each row, line_total equals qty x unit_price x (1 - discount_pct/100), rounded to 2 decimals;
  (b) each page subtotal equals the sum of the printed line totals on that page;
  (c) net_total equals the sum of the printed line totals of both pages;
  (d) tax_total equals the sum of the printed per-rate tax amounts in the totals block;
  (e) gross_total equals net_total plus tax_total.
At most one of these fails on an invoice. Never correct anything: in every field above, including line_total and the three totals, write the value as printed, even when the arithmetic is wrong.
When a check fails, set "inconsistency" to { "line": <printed row number for check (a), otherwise null>, "field": one of "line_total", "page_subtotal", "net_total", "tax_total", "gross_total", "printed": <the number as printed>, "computed": <the number the check gives, rounded to 2 decimals> }. When every check passes, set "inconsistency" to null.

Reply with the JSON object only.

The expected answer

{
  "invoice_number": "HP-2026-00731",
  "issue_date": "2026-04-08",
  "due_date": "2026-05-08",
  "currency": "EUR",
  "seller_name": "Harrowgate Print & Paper Ltd",
  "buyer_name": "Lismore Community College",
  "buyer_tax_id": "IE 6627104B",
  "line_items": [
    {
      "line": 1,
      "sku": "KB-9134",
      "qty": 7,
      "unit_price": 46.64,
      "discount_pct": 0,
      "tax_rate_pct": 13.5,
      "line_total": 326.48
    },
    {
      "line": 2,
      "sku": "MS-9164",
      "qty": 24,
      "unit_price": 21.35,
      "discount_pct": 12.5,
      "tax_rate_pct": 23,
      "line_total": 448.35
    },
    {
      "line": 3,
      "sku": "WC-6596",
      "qty": 13,
      "unit_price": 63.38,
      "discount_pct": 0,
      "tax_rate_pct": 23,
      "line_total": 823.94
    },
    {
      "line": 4,
      "sku": "CX-7521",
      "qty": 2400,
      "unit_price": 2.02,
      "discount_pct": 12.5,
      "tax_rate_pct": 23,
      "line_total": 4224
    },
    {
      "line": 5,
      "sku": "SH-8838",
      "qty": 10,
      "unit_price": 85.46,
      "discount_pct": 0,
      "tax_rate_pct": 23,
      "line_total": 854.6
    },
    {
      "line": 6,
      "sku": "LT-2453",
      "qty": 4,
      "unit_price": 18.46,
      "discount_pct": 0,
      "tax_rate_pct": 23,
      "line_total": 73.84
    },
    {
      "line": 7,
      "sku": "SN-3773",
      "qty": 21,
      "unit_price": 8.61,
      "discount_pct": 20,
      "tax_rate_pct": 23,
      "line_total": 144.65
    },
    {
      "line": 8,
      "sku": "LS-2284",
      "qty": 19,
      "unit_price": 30.88,
      "discount_pct": 2.5,
      "tax_rate_pct": 23,
      "line_total": 572.05
    },
    {
      "line": 9,
      "sku": "PT-2109",
      "qty": 4,
      "unit_price": 7.89,
      "discount_pct": 0,
      "tax_rate_pct": 13.5,
      "line_total": 31.56
    },
    {
      "line": 10,
      "sku": "MC-1179",
      "qty": 5,
      "unit_price": 325.41,
      "discount_pct": 20,
      "tax_rate_pct": 23,
      "line_total": 1301.64
    },
    {
      "line": 11,
      "sku": "SG-9791",
      "qty": 2400,
      "unit_price": 2.94,
      "discount_pct": 0,
      "tax_rate_pct": 13.5,
      "line_total": 7056
    },
    {
      "line": 12,
      "sku": "CT-5411",
      "qty": 13,
      "unit_price": 24.44,
      "discount_pct": 0,
      "tax_rate_pct": 23,
      "line_total": 317.72
    },
    {
      "line": 13,
      "sku": "DS-9630",
      "qty": 16,
      "unit_price": 128.98,
      "discount_pct": 0,
      "tax_rate_pct": 23,
      "line_total": 2063.68
    },
    {
      "line": 14,
      "sku": "SD-7259",
      "qty": 9,
      "unit_price": 91.67,
      "discount_pct": 10,
      "tax_rate_pct": 23,
      "line_total": 742.53
    },
    {
      "line": 15,
      "sku": "WF-1081",
      "qty": 8,
      "unit_price": 20.22,
      "discount_pct": 0,
      "tax_rate_pct": 23,
      "line_total": 161.76
    },
    {
      "line": 16,
      "sku": "WM-3815",
      "qty": 12,
      "unit_price": 12.22,
      "discount_pct": 2.5,
      "tax_rate_pct": 23,
      "line_total": 142.97
    },
    {
      "line": 17,
      "sku": "SP-3664",
      "qty": 3,
      "unit_price": 533.53,
      "discount_pct": 7.5,
      "tax_rate_pct": 13.5,
      "line_total": 1480.55
    },
    {
      "line": 18,
      "sku": "CW-7953",
      "qty": 12,
      "unit_price": 7.87,
      "discount_pct": 20,
      "tax_rate_pct": 23,
      "line_total": 75.55
    }
  ],
  "net_total": 20841.87,
  "tax_total": 3948.64,
  "gross_total": 24790.51,
  "inconsistency": {
    "line": 4,
    "field": "line_total",
    "printed": 4224,
    "computed": 4242
  }
}

What it tests. Every header field, each of the 18 rows and the three totals must match what is printed, to half a cent. The planted error is in row 4: the printed line total is 4224, while quantity times unit price less the discount gives 4242. The model has to copy the printed figure into the row, not the corrected one, and report the difference separately. A model that tidies the number up fails the row; one that copies everything and reports no inconsistency fails that part.

Support queue triage Cancel my plan and refund this year's renewal 9 parts checked

The system prompt carries the full support policy, about 31600 characters: 16 categories with an order of precedence, a priority matrix by plan and severity, 15 refund rules evaluated top to bottom, routing and escalation rules.

The letter

Subject: Cancel my plan and refund this year's renewal

Hello,

I have been on the Starter plan for a year. My annual plan, SKU SUB-STR-Y, renewed automatically on 11 November 2026 and my card was charged $120 for it under order QO-771204 (invoice INV-2026-11873). I did not mean to keep it: I moved to another tool in the summer, and nobody has opened Quillon since that charge, not me and not my assistant.

Please cancel the subscription so that it does not renew again, and refund the $120 for this renewal.

For reference, the first purchase a year ago was order QO-512330; that year was used and I am not asking about it.

Thank you,
Marta Ellison

The expected answer

{
  "category": "refund_request",
  "priority": "P3",
  "team": "billing_ops",
  "refund": {
    "eligible": true,
    "rule_id": "RF-11"
  },
  "order_id": "QO-771204",
  "sku": "SUB-STR-Y",
  "language": "en",
  "needs_escalation": false
}

What it tests. The letter asks for a cancellation and for money back, so the category is refund_request, which stands before cancellation in the policy's order of precedence. The charge is an automatic renewal: the 30-day rule for a first purchase of an annual plan (RF-10) does not apply, and the rule for an unused renewal within 7 days (RF-11) grants the refund. The order to act on is the renewal charge, not last year's purchase, which the letter names only as history.

Answer from documentation A rate limit that a later release changed 3 parts checked

The system prompt is the whole documentation bundle of a fictional API platform: 164 sections, about 152 thousand characters, with a reference, a changelog of six releases, plans and quotas, an SLA and a migration guide. The three excerpts above are the sections this question turns on; the model has to find them itself.

The question

What is the rate limit, in requests per minute, of `audience.segments.update` for an account on the Pro plan in v6? Answer with a single integer.

From the bundle: SEC-API-012

### [SEC-API-012] `audience.segments.update`
Endpoint: `PATCH /v6/audience/segments/update`
Available since: v1.
Status at launch: Active.
Plan availability at launch: Starter and above. Not supported on Free.
| Limit | Value at launch |
|---|---|
| Rate limit (requests per minute, Starter plan) | 600 |
| Maximum batch size (items per call) | not applicable |
| Maximum request payload (KB) | 1,024 |
| Unit cost (billing units per call) | 1 |

From the bundle: SEC-CHG-061

### [SEC-CHG-061] v6: limit scale
The rate limit of `audience.segments.update` was doubled. Sandbox keys received the change one day before production keys. The change applies to every API key on every plan.

From the bundle: SEC-PLN-001

### [SEC-PLN-001] Plan overview and rate multipliers
Values below are those of the launch plan table; later revisions are in the Changelog. The rate multiplier scales the Starter rate limits printed in the Reference: a method listed at 100 requests per minute allows 100 on Starter and 100 times the multiplier on any other plan. The multiplier applies to rate limits only. Exceptions are collected in [SEC-PLN-007].
| Plan | Monthly price | Included units per month | Rate multiplier |
|---|---|---|---|
| Free | $0 | 100,000 | 0.5 |
| Starter | $49 | 1,000,000 | 1 |
| Pro | $199 | 5,000,000 | 2 |
| Business | $799 | 20,000,000 | 4 |
| Enterprise | custom | 100,000,000 | 8 |

The expected answer

{
  "answer": 2400,
  "evidence": [
    "SEC-API-012",
    "SEC-CHG-061",
    "SEC-PLN-001"
  ]
}

What it tests. Three sections decide the answer and they sit thousands of lines apart. The Reference gives the launch value for the Starter plan (600). The v6 Changelog doubles it (1,200). The plan table multiplies Starter rates by 2 on Pro (2,400). The Reference entry is never edited after launch, so a reader who stops there answers 600, and one who misses the plan multiplier answers 1,200. The checker wants the number and the three sections it follows from.

Back-office agent with tools A broken mug, a return and a late address change 16 parts checked

The system prompt is a back-office policy of about 9200 characters in twelve sections: who may ask for what, refund amounts and fees, the refund ceiling, replacements, address changes, notes, escalation and the customer email. The model starts with the email only and has ten tools: search_orders, get_order, get_customer, check_stock, issue_refund, create_replacement, update_shipping_address, add_note, send_email, escalate. It has up to 30 steps.

The customer email

Case reference: CASE-5201
From: [email protected]
Received: 2027-03-15
Subject: Broken mug, a return, and a new address

Hello,

Three things, please.

One of the three ceramic mugs in order FH-31010 arrived with its handle snapped off. Could you send me a replacement for that one mug?

From the same order, the linen table runner is not the colour I hoped for. It is still sealed in its wrapper and I would like to return it for a refund.

And order FH-31044 should go to my new address instead: 9 Tannery Row, Eastmere, 30418.

Thank you,
Priya Venn

What the tools show when asked

FH-31010  status delivered, delivered 2027-03-06, shipping fee 6.00
  L1  MUG-118  Ceramic mug  x3  14.00 each
  L2  RUN-052  Linen table runner  x1  36.00 each
  L3  CND-007  Beeswax candle set  x2  12.50 each

FH-31044  status shipped, shipped 2027-03-14, shipping fee 5.00
  L1  TRY-031  Oak serving tray  x1  42.00 each

Stock: MUG-118: 5, RUN-052: 3, CND-007: 10, TRY-031: 2
Today: 2027-03-15

A correct run, call by call

get_customer {"email":"[email protected]"}
get_order {"order_id":"FH-31010"}
get_order {"order_id":"FH-31044"}
check_stock {"sku":"MUG-118"}
create_replacement {"order_id":"FH-31010","sku":"MUG-118","qty":1}
issue_refund {"order_id":"FH-31010","amount_cents":3240,"reason":"changed_mind","lines":[{"line_id":"L2","qty":1}]}
add_note {"order_id":"FH-31010","text":"CASE-5201: replacement mug sent; sealed table runner refunded minus the 10% restocking fee."}
escalate {"order_id":"FH-31044","queue":"shipping","summary":"CASE-5201: order already shipped; customer wants delivery to 9 Tannery Row, Eastmere 30418."}
add_note {"order_id":"FH-31044","text":"CASE-5201: address change requested after shipping, passed to the shipping team."}
send_email {"to":"[email protected]","case_ref":"CASE-5201","subject":"CASE-5201: orders FH-31010 and FH-31044","body":"Hello Priya, regarding CASE-5201. For order FH-31010 we are sending a replacement mug to the address on the order, free of charge. The table runner has been refunded: $32.40, which is its price of $36.00 minus our 10% restocking fee. Order FH-31044 has already shipped, so we could not change its address; your request with the new address has been passed to our shipping team.","outcomes":[{"order_id":"FH-31010","outcome":"replacement_sent","sku":"MUG-118"},{"order_id":"FH-31010","outcome":"refunded","amount_cents":3240},{"order_id":"FH-31044","outcome":"address_change_escalated"}]}

What it tests. Three requests, three different rules. The mug is in stock, so it is replaced and not refunded. The runner is a change of mind: its refund is the price less a 10% restocking fee, $32.40, and the shipping fee stays, because a changed-mind unit is part of the refund. The second order has already shipped, so its address must not be changed; the request goes to the shipping queue instead. Then one note on each order that was acted on and one email whose attachment lists exactly what was done. The checker reads the final state of the shop: the refund to the cent, the replacement, the untouched address, the notes, the escalation and the email. An agent that refunds the full $36.00, changes the address anyway or forgets a note fails.

Data analyst The five best-selling products of a quarter 6 parts checked

The model sees no data up front. It has two tools: get_schema, which lists tables, columns, row counts and three sample rows, and run_sql, which runs one SELECT and returns at most 50 rows. It has up to 20 steps.

The question

You are analysing the order database of Brightwave Goods, a fictional online shop. All figures are fictional.

Question: the five best-selling products, first quarter of 2026.

Scope: the order lines of paid orders placed from 2026-01-01 to 2026-03-31 inclusive. Refunds are not part of this question.

For each product report:
- units: the number of units sold in scope (the sum of the line quantities);
- gross_usd: the value of those lines in USD.

Answer with JSON only, exactly this shape: the five products with the most units, ordered by units from highest to lowest; products with equal units are ordered by sku from A to Z:
{"products":[{"sku":<string>,"units":<integer>,"gross_usd":<number>}, ...]}

Data dictionary, from the system prompt

Data dictionary of this database (it applies to every question asked about it):
- Timestamps (orders.placed_at, orders.shipped_at, refunds.refunded_at) are ISO 8601 instants with a UTC offset (Z or +/-HH:MM). Every day, month and period in a question is a UTC calendar date of those instants.
- An order is identified by orders.external_ref, a refund by refunds.refund_ref. Rows that share a reference are copies of one record and count once: keep the copy with the lowest id, together with its own order lines (refunds point to that copy).
- customers.is_test = 1 marks test customers. Their orders are never part of any figure.
- orders.status is paid, pending (not paid yet) or cancelled.
- customers.country is a two-letter ISO code stored in any letter case; report it in upper case. NULL means the country is unknown; report it as UNKNOWN.
- An order line is worth qty x unit_price x (1 - discount_pct / 100) in the order's currency; discount_pct NULL means no discount.
- refunds.amount is money returned against one order line (order_item_id), in the order's currency. A line can have several refunds.
- Currency: amounts are in orders.currency. To get USD, multiply by fx_rates.usd_per_unit of that currency for the UTC date on which the order was placed; when that date has no row, use the latest earlier row. USD needs no conversion and has no rows. Refunds use the rate of their order's placement date.
- Compute at full precision and round results to 2 decimals only at the end.

The expected answer

{
  "products": [
    {
      "sku": "SKU-0001",
      "units": 135,
      "gross_usd": 9205.99
    },
    {
      "sku": "SKU-0045",
      "units": 132,
      "gross_usd": 17290.48
    },
    {
      "sku": "SKU-0030",
      "units": 130,
      "gross_usd": 2781.49
    },
    {
      "sku": "SKU-0037",
      "units": 127,
      "gross_usd": 5330.86
    },
    {
      "sku": "SKU-0032",
      "units": 122,
      "gross_usd": 9148.65
    }
  ]
}

One query that returns it

WITH li AS (
  SELECT p.sku AS sku, i.qty AS qty, date(o.placed_at) AS d, o.currency AS cur,
         i.qty * i.unit_price * (1 - COALESCE(i.discount_pct,0) / 100.0) AS val
  FROM orders o
  JOIN customers c ON c.id = o.customer_id
  JOIN order_items i ON i.order_id = o.id
  JOIN products p ON p.id = i.product_id
  WHERE o.status = 'paid' AND c.is_test = 0 AND o.id IN (SELECT MIN(id) FROM orders GROUP BY external_ref)
), x AS (SELECT li.*, CASE WHEN li.cur = 'USD' THEN 1.0 ELSE (SELECT fx.usd_per_unit FROM fx_rates fx WHERE fx.currency = li.cur AND fx.rate_date <= li.d ORDER BY fx.rate_date DESC LIMIT 1) END AS rate FROM li)
SELECT sku, SUM(qty) AS units, ROUND(SUM(val * rate), 2) AS gross_usd
FROM x WHERE d BETWEEN '2026-01-01' AND '2026-03-31'
GROUP BY sku ORDER BY SUM(qty) DESC, sku ASC LIMIT 5

What it tests. The question itself is one GROUP BY. What decides pass or fail is the data dictionary in the system prompt: the tables hold test customers, duplicated order rows, unpaid and cancelled orders, orders in other currencies and discounts stored as NULL. A query that ignores any one of these returns a different top five or different sums, and the checker compares every number: units exactly, dollars within five cents.

§ 09 / METHOD

How it is measured

Same tasks for everyone

Every model gets the same 82 tasks with the same instructions, in the same order. Nothing is tuned per model: no prompt changes, no temperature, no reasoning setting. The model runs the way its maker ships it, at its default reasoning effort, which is printed next to its name.

The maker's own API

Requests go through OpenRouter pinned to the endpoint of the company that makes the model, with fallbacks off. DeepSeek V4.1 Flash is called on its maker's own API instead, because OpenRouter sells it only through other hosts. A model we could reach only through a third-party host is not in the table.

A program decides pass or fail

Every task has a reference answer computed by code, and a checker compares the model's answer part by part. No model grades another model, and there is no partial credit: one wrong field fails the task.

The bill

Each response reports its tokens: input at full price, input read from cache, cache writes, output with reasoning inside. We multiply each by the vendor's list price from our price index and add them up. Alongside we keep what the request was charged, and flag a run where the two part by more than 5%. DeepSeek V4.1 Flash ran on its maker's own API, which returns no charge per request: there the charge is the drop in the account balance over the run, read to the cent once it settled.

The index: tasks done right per dollar

The cost of a clean result is the bill plus 10 cents for every failed task: a failed task is not free, someone has to catch it and do it again. The index is the 82 tasks divided by that cost. Ranked by bill alone, or by bill per passed task, the cheapest model comes first even when it fails a tenth of the work.

Why 10 cents

It is an assumption, so the table lets you change it from zero to a dollar. We did not set it higher because each row is one run and a pass count moves by a few tasks between runs: at a dollar per failure the ranking would mostly rank that chance. At 10 cents a gap under $0.40 is a tie.

Caching as the vendor documents it

Long shared prompts are how real workloads look, so caching is on, the way each vendor says to use it: automatic where it is automatic, with cache markers where the vendor requires them. A single one-off request gets no marker.

A cold start for every run

Each run starts with a random line at the top of the system prompt, so it cannot read a cache that an earlier run paid for. Inside a run the cache works normally.

One run, the latest one

The table shows the latest finished run of each model on the current task set. We do not run a model several times and keep the best. A run cut short by an error is not published.

When a task turns out to allow two readings, we rewrite it and run only that task again on each model; the new result replaces the old one and the rest of the run stays as it was. Support queue triage task sq-037 was rewritten and rerun on November 21, 2026 by 16 of the 17 models. MiniMax M3.1 Flash keeps its result on the first version: MiniMax no longer serves this model on OpenRouter, and we run a model only on its maker's endpoint.

A cap, not a handicap

Output is capped at 32,000 tokens per request and loops at 20 to 30 steps. The caps are a spending guard set well above what the tasks need; a model that hits one fails that task.

§ 10 / LIMITS

What this does not tell you

§ 11 / RUN LOG

The runs behind the table

Model Run date Served by Reasoning effort Caching Input tokens Output tokens Bill on run day Charged
Hunyuan Hy4 Flash Tencent no named levels reasoning on by default Vendor's automatic caching 1.77M 230K $0.139 $0.069
MiMo-V2.6-Flash Xiaomi no named levels reasoning on by default Vendor's automatic caching 1.97M 240K $0.144 $0.144
Muse Ember Meta max default, cannot be turned off · of max, xhigh, high, medium, low, minimal Vendor's automatic caching 2.63M 283K $0.303 $0.303
Claude Haiku 5.5 Anthropic medium default · of max, xhigh, high, medium, low, none Cache markers on shared prompts and on tool loops 1.86M 122K $0.112 $0.112
GLM-5.3-Flash Z.AI max default, cannot be turned off · of max, high, low Vendor's automatic caching 1.55M 234K $0.257 $0.257
DeepSeek V4.1 Flash DeepSeek high default · of max, high, low, none Vendor's automatic caching 1.58M 174K $0.265 $0.130
GPT-6.1 Terra OpenAI medium default · of max, xhigh, high, medium, low, none Vendor's automatic caching 1.29M 48K $0.546 $0.546
GLM-5.3-FlashX Z.AI max default, cannot be turned off · of max, high, low Vendor's automatic caching 1.39M 171K $0.407 $0.407
MiniMax M3.1 Flash Minimax max default, cannot be turned off · of max, xhigh, high, medium, low Vendor's automatic caching 3.07M 263K $0.307 $0.307
GPT-6 Luna OpenAI medium default · of max, xhigh, high, medium, low, none Vendor's automatic caching 1.47M 80K $0.080 $0.080
Qwen3.7 Plus Alibaba no named levels reasoning on by default Vendor's automatic caching 2.18M 248K $0.705 $0.564
Step 5 Preview StepFun medium default, cannot be turned off · of high, medium, low Vendor's automatic caching 1.80M 214K $1.15 $1.15
Claude Sonnet 5.5 Anthropic high default, cannot be turned off · of max, xhigh, high, medium, low Cache markers on shared prompts and on tool loops 1.75M 79K $1.59 $1.59
Kimi K3.1 Pro Moonshot AI max default · of max, high, low, none Vendor's automatic caching 1.44M 141K $2.17 $2.96
Muse Spark 1.3 Meta medium default, cannot be turned off · of max, xhigh, high, medium, low, minimal Vendor's automatic caching 2.31M 198K $2.60 $2.60
Gemini 4 Hydrogen Google AI Studio medium default, cannot be turned off · of xhigh, high, medium, low Vendor's automatic caching 1.78M 269K $4.09 $4.09
Gemini 3.5 Flash-Lite Google AI Studio minimal default, cannot be turned off · of high, medium, low, minimal Cache marker on shared prompts 1.61M 56K $0.313 $0.313

"Reasoning effort" is the level each run used. We send no reasoning setting, so it is the default of the model's API route as the public model catalog listed it on the run day; the line under it gives the levels the route accepts. For DeepSeek V4.1 Flash, run on the maker's own API, it is the default the maker's API documentation states. "Bill on run day" uses the list prices of that day; "charged" is what OpenRouter billed for the same requests. DeepSeek V4.1 Flash ran on its maker's own API, which returns no charge per request: there the charge is the drop in the account balance over the run, read to the cent once it settled. Release 2026-11: runs from 2026-11-14 to 2026-11-21.

§ 12 / REUSE

Use these numbers, with a link

The results, charts and text of AI Cost Bench are licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Quote them, re-chart them, republish them, commercially too, without asking. The condition is credit: name AI Cost Bench by AI//COST and link to this page. If you redraw a chart, keep the credit on the image and say what you changed. The vendor marks on the index chart belong to their owners and only show whose model each bar is: the licence covers our numbers and drawings, not the marks.

Put the index chart on your site

Paste one of these where the chart should appear. The picture is served from this site and redrawn with every update, so it follows the ranking; the caption under it is the credit the licence asks for. Pick the picture that suits your page: dark, or light for a white or cream background.

Embed · HTML
<figure>
  <img src="https://aicost.tools/benchmarks/ai-cost-bench/index-chart.svg" alt="AI Cost Bench Index: work tasks done right per dollar, by AI model" width="1200" height="675" style="max-width:100%;height:auto" loading="lazy">
  <figcaption>Source: <a href="https://aicost.tools/benchmarks/ai-cost-bench/">AI Cost Bench by AI//COST</a></figcaption>
</figure>
Embed · Markdown
![AI Cost Bench Index: work tasks done right per dollar, by AI model](https://aicost.tools/benchmarks/ai-cost-bench/index-chart.svg)

Source: [AI Cost Bench by AI//COST](https://aicost.tools/benchmarks/ai-cost-bench/)
AI Cost Bench Index as the dark picture the embed code shows
What the embed shows. The "Download PNG" button above the chart saves the picture in the theme you are reading this page in.

Credit lines

Credit line · text
Source: AI Cost Bench by AI//COST, https://aicost.tools/benchmarks/ai-cost-bench/ (CC BY 4.0)
Credit line · HTML
Source: <a href="https://aicost.tools/benchmarks/ai-cost-bench/">AI Cost Bench by AI//COST</a> (<a href="https://creativecommons.org/licenses/by/4.0/" rel="license">CC BY 4.0</a>)
Credit line · Markdown
Source: [AI Cost Bench by AI//COST](https://aicost.tools/benchmarks/ai-cost-bench/) ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/))
Cite as
AI//COST (2026). AI Cost Bench, release 2026-11. https://aicost.tools/benchmarks/ai-cost-bench/. Retrieved 2026-11-21. CC BY 4.0.
BibTeX
@misc{aicostbench202611,
  author = {{AI//COST}},
  title = {AI Cost Bench, release 2026-11},
  year = {2026},
  url = {https://aicost.tools/benchmarks/ai-cost-bench/},
  note = {Retrieved 2026-11-21. Licensed under CC BY 4.0}
}

AI Cost Bench FAQ

Answers derived from the 17 runs on this page and the current price index.

Q · 01

Which AI model is the best value on AI Cost Bench?

+
Hunyuan Hy4 Flash. It tops the index at 343 tasks done right per dollar, because it has the lowest cost of a clean result, $0.239: a $0.139 bill plus 1 failed task at 10 cents each. Next is MiMo-V2.6-Flash at $0.244. A gap under $0.40 is inside the spread between repeat runs, so MiMo-V2.6-Flash, Muse Ember, Claude Haiku 5.5, GLM-5.3-Flash and DeepSeek V4.1 Flash are level with it. The order depends on what a wrong answer costs you. Below 0.4 cents per failed task GPT-6 Luna comes first, from 0.4 cents to 2.7 cents Claude Haiku 5.5, from 2.7 cents to 16.4 cents Hunyuan Hy4 Flash, above 16.4 cents Muse Ember.
Q · 02

Is AI Cost Bench a test of how smart a model is?

+
No. Capability benchmarks such as ARC-AGI-2, Humanity's Last Exam and SWE-Bench Pro ask how hard a problem a model can solve. AI Cost Bench asks what everyday work costs: 82 tasks of 6 kinds, the support letters, documentation questions, invoices, back-office cases, SQL reports and bug fixes a product hands to a model over and over. Getting the work right still counts, because every failed task adds 10 cents to the cost, so the index ranks the cost of work done right. Muse Ember, Kimi K3.1 Pro and Gemini 4 Hydrogen all passed all 82, with bills of $0.303, $2.17 and $4.09. A model that passes all 82 handles this kind of work; it can still rank low on a hard reasoning test.
Q · 03

How is the AI Cost Bench Index calculated?

+
The index is the number of tasks a dollar gets done right: the 82 tasks divided by the cost of a clean result. The cost of a clean result is the bill for the 82 tasks plus 10 cents for every task the model failed. The bill alone, or the bill divided by tasks passed, puts the cheapest model first even when it fails a tenth of the work, because bills differ 51x and pass counts by a few percent. Pricing a failure fixes that: a failed task has to be caught and done again. 10 cents is our assumption, and the switch above the table changes it from zero to a dollar. We did not publish a higher default because every row is one run: at a dollar per failure a chance swing of a few tasks would decide the ranking.
Q · 04

Which AI model is cheapest on AI Cost Bench?

+
GPT-6 Luna: $0.080 for all 82 tasks, with 72 passed. Next is Claude Haiku 5.5 at $0.112 with 80 passed. The most expensive run was Gemini 4 Hydrogen at $4.09.
Q · 05

Which model gets the most tasks right?

+
Muse Ember, Kimi K3.1 Pro and Gemini 4 Hydrogen passed all 82 tasks. Every row is a single run. When we ran two models six times each while calibrating the harness, their pass counts moved by 4 and by 8 tasks, so read a gap of 4 tasks or less as a tie. The cheapest model within that distance of the top is Claude Haiku 5.5 at $0.112.
Q · 06

Why can a model with a lower price per token end up with a higher bill?

+
Because the bill is price times tokens, and models spend very different numbers of tokens on the same task. Two things move it most. Reasoning: GLM-5.3-Flash spent 80% of its output tokens on reasoning, Gemini 3.5 Flash-Lite 0%, and reasoning is billed at the output rate. Caching: DeepSeek V4.1 Flash read 90% of its input from cache at a fraction of the input price, Gemini 4 Hydrogen 23%.
Q · 07

How is the bill calculated?

+
From the token counts the vendor's API returns with every response: input at full price, input read from cache, cache writes and output (reasoning included), each multiplied by the vendor's list price in our price index. We also record what the request was actually charged; where the two differ by more than 5% the row says so. DeepSeek V4.1 Flash ran on its maker's own API, which returns no charge per request: there the charge is the drop in the account balance over the run, read to the cent once it settled. Subscriptions, batch and flex discounts are not applied.
Q · 08

What reasoning effort were the models run at?

+
At each model's default. We send no reasoning setting, so every model reasons as much as its API does out of the box, and that differs from model to model. Muse Ember, GLM-5.3-Flash, GLM-5.3-FlashX, MiniMax M3.1 Flash and Kimi K3.1 Pro ran at max, DeepSeek V4.1 Flash and Claude Sonnet 5.5 at high, Claude Haiku 5.5, GPT-6.1 Terra, GPT-6 Luna, Step 5 Preview, Muse Spark 1.3 and Gemini 4 Hydrogen at medium, Gemini 3.5 Flash-Lite at minimal; Hunyuan Hy4 Flash, MiMo-V2.6-Flash and Qwen3.7 Plus have no named effort levels. The level is shown in brackets on the index chart and in the run log, with the levels each model accepts. It is the default of the model's API route as the public model catalog listed it on the run day. For DeepSeek V4.1 Flash, run on the maker's own API, it is the default the maker's API documentation states.
Q · 09

How many times was each model run?

+
Once. The table shows the latest finished run of each model on the current task set, never the best of several. Bills repeat closely from run to run; pass counts do not, which is why a small gap in tasks passed is not a ranking.
Q · 10

Are the tasks public?

+
The six kinds of task and the way each is checked are described on this page. The 82 tasks themselves stay private so they do not end up in training data, which would turn the benchmark into a memory test. One sample task of each kind, written separately and never part of the 82, is shown in full with its expected answer. Every task has a reference answer that the checker itself accepts.
Q · 11

Can I republish these results?

+
Yes. The results, charts and text of AI Cost Bench are licensed under Creative Commons Attribution 4.0 (CC BY 4.0): copy, quote, re-chart and republish them, commercially too, as long as you credit AI Cost Bench by AI//COST with a link to this page. The licence link, a ready credit line and the data files are in the reuse section.

One weekly digest. Zero noise.

One weekly digest · No spam, ever · Unsubscribe in one click