What a token is
A language model does not see letters or words. Before your prompt reaches it, a tokenizer cuts the text into pieces drawn from a fixed vocabulary, and each piece becomes a number the model works with. A piece can be a whole common word, part of a longer word, a single character, or a punctuation mark. The model then writes its answer the same way, one token at a time.
OpenAI's rule of thumb for English is that one token is about four characters, or three-quarters of a word, so 100 tokens come to roughly 75 words. Google gives a similar figure for Gemini: about four characters per token, and 100 tokens for roughly 60 to 80 English words. Both vendors call these estimates, not counts.
Small details move the count. OpenAI points out that red, Red and red with a leading space are different text and can be encoded differently, and that a token ID from one encoding means nothing in another. That is why no rule of thumb replaces counting with the tokenizer of the model you will actually call.
Why the same text gives different token counts
Each vendor trains its own tokenizer, and a vendor can change it between model generations. The same paragraph therefore costs a different number of tokens depending on where you send it — which matters, because the price you compare is per token, not per word.
The change can be large. Anthropic documents that Claude Opus 4.7 and later models use a newer tokenizer, and that the same input text produces approximately 30 percent more tokens than on earlier models. Its advice is to recount prompts against the model you plan to use rather than reuse counts measured on an older one.
Content type matters as much as the vendor. Code and structured data such as JSON pack fewer characters into each token than prose, so a page of code is more tokens than a page of English of the same length. Other languages split differently again; OpenAI notes that non-English text can have a different relationship between characters, words and tokens.
In our indexSame text, different token counts
| Tokenizer | Prose chars/token | Code | JSON | Words/token |
|---|---|---|---|---|
| MiniMax | 5.33 | 3.33 | 2.69 | 0.881 |
| OpenAI | 5.28 | 3.47 | 2.71 | 0.872 |
| DeepSeek | 5.26 | 3.26 | 2.58 | 0.869 |
| GLM (Zhipu) | 5.25 | 3.36 | 2.69 | 0.867 |
| Qwen (Alibaba) | 5.23 | 3.19 | 2.24 | 0.863 |
| Hunyuan (Tencent) | 5.23 | 3.34 | 2.49 | 0.863 |
| Baichuan | 5.23 | 3.34 | 2.49 | 0.863 |
| Doubao (ByteDance) | 5.20 | 3.16 | 2.22 | 0.858 |
| Mistral | 5.09 | 3.28 | 2.42 | 0.840 |
| Yi (01.AI) | 5.06 | 2.71 | 2.00 | 0.835 |
Measured on 2026-06-10 with each vendor's own published tokenizer on one fixed English corpus (436 words of prose plus code and JSON samples). Anthropic and Google publish no tokenizer, so they are not in this table. How we collect prices.
How tokens show up on a bill
LLM APIs price tokens per million, and they price them by direction. Input tokens are everything you send — the system prompt, the conversation so far, any documents and tool definitions. Output tokens are what the model writes back, and they cost more: Claude Sonnet 5.5 lists $2 per million input tokens against $10 per million output tokens. The split is covered in input vs output tokens.
Two more categories sit on most invoices. Cached input tokens are input the provider has already processed and reuses at a lower rate. Reasoning tokens are the hidden thinking a reasoning model does before it answers; OpenAI states they are not shown as answer text but are billed as output tokens.
Images, audio and video are converted into tokens too. Google documents that a Gemini image of 384 pixels or less on both sides counts as 258 tokens, larger images are tiled into 768×768 pieces at 258 tokens each, and audio runs at 32 tokens per second. Other vendors use their own formulas, so a picture can be cheap on one API and expensive on another.
Worked examplePricing a support assistant by the token
10,000 requests a month on Claude Sonnet 5.5, priced at its current list rates, checked on 2026-10-09.
Input is the same in both columns — a system prompt plus the question. Every dollar of difference is output, which carries the higher rate.
Change the workload or the model in the LLM API cost calculator.
How to count tokens before you pay
Every major vendor gives you a way to count without sending the real request:
- OpenAI publishes the
tiktokenlibrary for plain text and an input-token counting API for complete Responses requests, which includes the formatting tokens around messages, tools and files. - Anthropic offers a token counting endpoint that is free to use, subject to requests-per-minute limits by usage tier. Anthropic labels the result an estimate that can differ from the billed count by a small amount.
- Google exposes
count_tokensfor the input, and returns input, output, thinking and cached token totals with each response.
Counting the input does not predict the output: OpenAI's help page says so explicitly. To budget a workload, measure the output on representative requests, then price both sides. Our token counter does the first part in your browser and the LLM API cost calculator does the second.
Mistakes that inflate a token bill
- Reusing counts across models. A prompt measured on one tokenizer is a different size on another — by roughly 30 percent across Anthropic's own tokenizer change.
- Forgetting what rides along. The system prompt, earlier turns and tool definitions are input on every call, not just the user's question.
- Ignoring hidden output. A reasoning model can bill far more output than the answer you see.
- Comparing price per token instead of price per task. OpenAI notes that a lower price per million tokens does not guarantee a lower total cost, because models tokenize differently and write different amounts.