Five rules govern everything we publish. They are locked into the methodology — changing them requires a quarterly review with a written rationale, not an editorial decision. The rules below are the surface; the rest of this document is how they're operationalized.
Verify before asserting
Anything that is not a well-known constant must be verified at the time of publication. Prices change, APIs deprecate, benchmark scores update. We don't quote from memory or training data. Each claim has a timestamp and a source.
Score before ranking
Rubrics are locked before testing. We define criteria + weights publicly, then run tests. Scores for one candidate are not visible while another is being scored — each pass is blind to the others. Rankings are a function of scores, not preferences.
Honesty over diplomacy
If a tool is bad at something, we say so. Every brand profile has a "Don't use it for" section. If we'd recommend a competitor for a use case, we link to the competitor — even when our affiliate relationship is with the subject.
Sponsorship quarantined
Editorial rankings are not for sale. Sponsored slots exist on some pages but are visually quarantined (dashed warm border, "ⓘ SPONSORED" tag). They cannot influence editorial ranking, scoring, or "best of" picks.
Corrections are public
When we get something wrong, we log it openly. The corrections page lists every correction with date, page, what was wrong, and what's now correct. We don't silently edit. We don't gaslight readers about what we used to say.
Methodology is versioned
This document is v2.4. Major changes ship as version bumps with a changelog. You can compare v2.0 and v2.4 to see exactly what shifted in our approach over time. No silent rewrites of how we work.
If you find a violation of any rule above, email [email protected] with the URL and the issue. We respond within 5 business days and either correct, push back with reasoning, or escalate to a public correction. Every correction we issue is logged at /corrections/ — that page is the canonical source of truth for our track record.
Pricing for 264 LLM models across 20 providers is verified manually against each vendor's own canonical pricing page. No scrapers. No automated billing-API pulls. No third-party aggregators. The trade-off is honest: slower updates than a scraping pipeline would deliver, but every number has a human who checked the source URL on the date stamped next to the price.
Canonical source identified
For each model, the vendor's own canonical pricing page is the source of truth (e.g., anthropic.com/pricing, openai.com/api/pricing). Press releases, blog posts, and third-party aggregators are never used as source — they're often stale or out of date the moment a vendor ships a change.
Manual verification
Each price is fetched from the canonical URL via WebFetch, read against the published rate card, and the input rate, output rate, cache rate, and any tier thresholds are recorded by hand into the model's JSON record. One human, one source URL, one verification date — no parser layer between the vendor page and the published number.
Diff against snapshot
The new reading is compared against the previous published snapshot. Any change triggers a re-read of the full pricing page (not just the row that moved) to make sure the rest of the rate card didn't change at the same time. Mismatches are reconciled before the new price is published.
Publish + log
Approved changes propagate to all dependent pages on the next build. The price-history table records the change with timestamp, old value, new value, and the source URL that was verified. Every price on the site has a verifiable provenance trail. Target latency from a vendor announcement to a published correction is under 24 hours; trigger is a vendor-blog signal or a reader-reported discrepancy.
| Source type | Example | Refresh | Verification |
|---|---|---|---|
| Vendor pricing page | Manual WebFetch read of anthropic.com/pricing, openai.com/api/pricing, etc. | On change | ✓ Manual |
| Vendor announcement | Official blog/social signal triggers immediate re-read of the canonical page | ~24h target | ✓ Manual |
| Currency exchange | European Central Bank reference rate feed (planned — Phase 2, currency display is USD-only today) | — | ~ Planned |
| Country tax/VAT | Static config sourced from official government pages (planned — Phase 2, country pages not yet shipped) | — | ~ Planned |
| Third-party aggregators | Listed as not used as source; only used as a sanity cross-check | — | ✗ Not source |
For every coding agent (Claude Code, Cursor, Aider, etc.) that gets a profile, we run all 500 SWE-bench Verified tasks ourselves on a clean Linux box with stock configurations — no custom scaffolding, no prompt tuning. This costs us ~$170-500 per agent in API spend but is the only way to publish numbers we trust.
For general-purpose agents (Manus, Devin) we run τ-bench and GAIA. For browser agents (Browser Use, Manus web mode) we run WebArena. The test harness, configurations, and full result spreadsheets are public.
| Test type | What we run | Per agent | Reproducibility |
|---|---|---|---|
| Coding · SWE-bench | All 500 Verified tasks, stock config, 14 hour runtime cap | ~$170-500 | ✓ Public test set |
| Tool use · τ-bench | Multi-turn API workflows · airline + retail subsets | ~$80-200 | ✓ Public |
| General · GAIA | Multi-modal reasoning · level 1-3 tasks | ~$60-150 | ✓ Public |
| Browser · WebArena | Web automation · admin/social/shopping subsets | ~$50-120 | ✓ Public |
| Self-reported scores | Vendor scores accepted only with caveat label "self-reported" | — | ~ Marked |
Vendor-published benchmark scores often use custom scaffolding, optimized prompts, or larger context windows than typical users get. Self-reported numbers can be 5-15 points higher than what users experience with stock config. We run tests the way users actually use the agent — and label any score we accepted from vendor as "self-reported."
We track 8 active benchmarks across the AI agents cluster. For each, we publish a standalone benchmark page with our verification status, methodology, last run date, and full leaderboard. Self-reported scores from vendors are accepted with explicit caveats; reproduced scores get the "verified by us" tag.
- SWE-bench Verified — 500 GitHub issue tasks, all reproduced
- τ-bench — multi-turn tool use, airline + retail subsets reproduced
- GAIA — general assistant tasks, level 1-3 reproduced; level 4 self-reported
- WebArena — web automation, admin/social/shopping subsets reproduced
- HumanEval — saturated; we accept self-reported with skepticism
- LiveCodeBench — contest problems, monthly refresh
- SWE-bench Multimodal — visual context tasks, partial reproduction
- Aider polyglot — multi-language, reproduced via Aider's harness
Every benchmark page links to the original paper, the public test set, and our reproduction scripts. If a vendor publishes a number that diverges >10% from our reproduction, we annotate the discrepancy publicly rather than just accepting their figure.
Editorial rankings (Top 10 lists, Best-of picks) follow a strict process: rubric locked → tools scored independently → outliers re-reviewed → final ranking computed. Sponsorship cannot move a tool up or down. Affiliate relationships exist but are disclosed per-link.
Rubric locked
For each ranking, we define 6-8 criteria + explicit weights (e.g., Aesthetic 25%, Adherence 20%, Cost 15%). Rubric is published before testing starts. Any change to rubric requires version bump.
Pass one — rubric scoring
Founder scores every candidate against the locked rubric, criterion by criterion. Each candidate is scored in isolation; the previous candidate's scores are not visible while scoring the next one. Sponsor status is not visible at this stage. One scorer, one rubric, no shortcuts.
Pass two — checklist sweep
A separate checklist pass verifies the scores against the underlying evidence: each criterion must point at a specific source, screenshot, or test run. Anything not anchored to evidence gets flagged for re-scoring. This catches the "I just felt it" kind of mistake.
Pass three — cool-off review
After a 48-hour cool-off, the founder re-reads the entire ranking from a fresh head. Score adjustments at this stage are rare but logged with the reason. Final 0-100 score is weighted per rubric and summed; ranking is deterministic from the score. If a sponsor lands in #1 by the math, that's the math. One person, three passes, no manual override.
For 100+ countries, we publish localized pricing pages with currency, VAT/GST, payment methods, and availability. Base USD price comes from vendor; conversion uses ECB hourly rate; tax is per-country static (reviewed quarterly); payment availability is verified manually with country-specific test accounts where possible.
Tax rates are sourced from official government tax authority pages — not third-party tax-rate APIs that often lag actual policy. Country DB is reviewed each quarter for VAT changes; emergency updates happen for rate changes (e.g. Germany VAT change in 2020-2021).
VAT logic for major countries is reviewed by a country-local accountant (currently DE/FR/UK reviewed by named accountants — listed in each country page byline). Where we don't have local accountant review, the country page carries a "tax logic not locally verified" caveat.
Every calculator uses verifiable math — no black-box estimation. The formula is printed on each calculator page, every rate comes from the same vendor-verified pricing snapshot as the rest of the site, and all numbers on a calculator page (presets, worked examples, comparison tables) are recomputed from that snapshot on every rebuild — nothing is hand-typed.
The LLM API cost calculator models the levers that actually move bills: prompt-cache hit rates with vendor cache-write premiums, batch-tier rates, and hidden reasoning tokens billed at the output rate. Default assumptions are shown in the UI, not hidden, and every input can be tweaked.
Token counting is labeled with one of three accuracy tiers, on every row. exact — OpenAI models counted in your browser with the o200k_base BPE, the same encoding the API uses. cal — vendors whose token counts use a chars-per-token calibration we measured on the vendor's own published tokenizer (downloaded from their official Hugging Face repos; measurement date shown on the token counter). est — vendors with no public tokenizer (Claude, Gemini, Grok, Kimi), estimated from vendor documentation. Where a closed API model is calibrated via the vendor's open-weight sibling, the page says so.
When vendors reprice or release new tokenizers, the snapshot and calibrations are re-verified and the whole section recomputes on deploy. Found math that looks wrong? Report it — calculator errors are treated as corrections, not feedback.
For "Can X do Y?" pages (e.g., "Can Claude Code work offline?"), we run the test ourselves, capture the actual output, and label answers with verification status. The tested version, date, and exact command/prompt used are shown on every capability page.
Where the answer is "no," we always provide workarounds — the alternative path or tool that does support the capability. We don't just confirm a limitation and walk away. The capability question template's "Alternative path" section is required for any "no" answer.
For every install guide, the author installs the tool on a clean machine on each platform we cover (macOS, Linux, Windows). Steps are recorded as the install runs; expected output is captured from actual terminal output, not described from memory. Tested-with version is logged in the byline.
Guides are re-verified quarterly or whenever a tool ships a new major version. Common errors come from our own install logs plus support-channel monitoring on each tool's Discord/forum/Stack Overflow tag — we add troubleshooting items as we encounter or hear about them.
Sponsored slots exist on some pages. They are visually quarantined (warm-amber dashed border, "ⓘ SPONSORED" tag, never inside editorial ranking blocks) and cannot affect editorial scoring. Sponsorship is sold at fixed published rates.
Affiliate relationships are separate from sponsorship. Many tools we recommend have affiliate programs we participate in. Each affiliate link is marked with the ↗ symbol. Affiliate revenue funds our testing budget. Affiliate status cannot move a tool up or down in editorial rankings — and we cite tools without affiliate programs (DALL-E, Imagen, OpenAI direct API) on their merits when they win on rubric.
Top-N positions, "best of" picks, scoreboard rankings, brand profile verdict, comparison verdicts. Sponsors can buy quarantined slots in clearly-labeled sponsored sections. They cannot buy editorial position. We have turned down 6-figure offers on this; the policy is non-negotiable.
When we get something wrong, we log it. The corrections page lists every correction with: date issued, page affected, what was wrong, what's now correct, who flagged it. We don't silently edit. We don't claim things were always correct that were not.
Severity levels: minor (typo, broken link) → fixed silently with timestamp on page. Material (incorrect price, wrong benchmark score, wrong feature claim) → corrected page + entry in /corrections/ + email reply to whoever flagged it. Severe (incorrect ranking, retracted claim) → corrected page + corrections log + apology paragraph in next quarterly editorial review.
The canonical record of every correction we've issued lives at /corrections/ — each entry includes the original claim, the corrected claim, the date, and the page affected.
Founder signs off on every published claim. AI assists with drafting and formatting; the human author makes every editorial decision and verifies every fact. As the desk scales, additional named reviewers will appear here with verifiable handles — second-pair-of-eyes hires are welcome.
Yaroslav VikharievFounder
Founder of AI Cost & Tools Hub. Editorial sign-off on every pricing claim. Verifies prices against the vendor's own canonical pricing page; leaves fields blank rather than fabricating.
Every change to this methodology is logged. The page you're reading is v2.4; you can compare any two versions by passing ?v=X.Y query parameters.