How a tokenizer works
A model's vocabulary is a fixed list of text pieces, each with a number. The tokenizer's job is to express any input as a sequence of those pieces. Frequent words become a single token; rarer words are assembled from several sub-word pieces; anything truly unusual falls back to individual bytes, so no text is ever unrepresentable.
Most current tokenizers use some form of byte-pair encoding, which builds the vocabulary by repeatedly merging the most frequent pairs of symbols in a training corpus. OpenAI's tiktoken README describes the result in three properties: it is reversible and lossless, so tokens convert back into the original text; it compresses the text; and it lets the model see common sub-words. The same process runs in reverse on output — the model produces token numbers and the tokenizer turns them back into text.
Why tokenizers disagree
A vocabulary reflects the data it was built from. A tokenizer trained on lots of code merges common code patterns into single tokens; one trained heavily on English prose does the same for English. Vendors also choose different vocabulary sizes. The result is that one paragraph can be a noticeably different token count on two models — and since the price is per token, a model with a lower list rate can still cost more for your text.
Tokenizers also change between model generations. Anthropic documents that Claude Opus 4.7 and later models use a newer tokenizer that produces approximately 30 percent more tokens for the same input than earlier models, and tells developers to recount prompts on the model they plan to use. A migration that keeps the per-token price flat can still raise the bill.
Content type changes the ratio as much as the vendor does. In our measurements, code and JSON pack noticeably fewer characters into each token than English prose on every tokenizer we could load — the table below shows the figures.
In our indexSame text, different token counts
| Tokenizer | Prose chars/token | Code | JSON | Words/token |
|---|---|---|---|---|
| MiniMax | 5.33 | 3.33 | 2.69 | 0.881 |
| OpenAI | 5.28 | 3.47 | 2.71 | 0.872 |
| DeepSeek | 5.26 | 3.26 | 2.58 | 0.869 |
| GLM (Zhipu) | 5.25 | 3.36 | 2.69 | 0.867 |
| Qwen (Alibaba) | 5.23 | 3.19 | 2.24 | 0.863 |
| Hunyuan (Tencent) | 5.23 | 3.34 | 2.49 | 0.863 |
| Baichuan | 5.23 | 3.34 | 2.49 | 0.863 |
| Doubao (ByteDance) | 5.20 | 3.16 | 2.22 | 0.858 |
| Mistral | 5.09 | 3.28 | 2.42 | 0.840 |
| Yi (01.AI) | 5.06 | 2.71 | 2.00 | 0.835 |
Measured on 2026-06-10 with each vendor's own published tokenizer on one fixed English corpus (436 words of prose plus code and JSON samples). Anthropic and Google publish no tokenizer, so they are not in this table. How we collect prices.
Counting when the tokenizer is not public
OpenAI publishes its tokenizer, and open-weight models ship theirs alongside the weights, so their counts can be reproduced exactly offline. Anthropic and Google do not publish theirs. For those models the vendor's counting endpoint is the authority: Anthropic's token-counting API is free and rate-limited by usage tier, and Google exposes count_tokens. Our token counter is exact for OpenAI's encoding and uses measured characters-per-token calibrations for the other vendors, which is an estimate and is labelled as one.