Where the transformer came from
The transformer was introduced in June 2017 in “Attention Is All You Need”, a paper by eight researchers working at Google Brain and Google Research, one of them from the University of Toronto. Language models of the time read text one word after another through recurrent networks, which made them slow to train because each step had to wait for the last. The paper dropped recurrence entirely and built the network from attention alone.
The target was machine translation, and the results were strong: better scores than any previous system on English-to-German, and a new record on English-to-French after three and a half days of training on eight GPUs — which the authors described as a small fraction of what earlier top models had cost to train. Within a few years the same design, scaled up, became the basis for the language models now sold per token.
How attention works, in plain terms
Think of each token in a sentence asking a question of all the others: which of you matter for what I mean here? In “the bank raised its rates”, the token bank pays attention to rates and settles on the financial sense. Mechanically, every token is turned into three vectors — one that asks, one that answers, one that carries content — and each token's output is a weighted mix of the others' content, weighted by how well their answers match its question.
A transformer runs many of these attention operations side by side, so different ones can track grammar, topic or reference, and stacks dozens of layers so later ones build on earlier ones. Because attention by itself ignores word order, position information is added to each token on the way in.
Why you are billed per token
The token is the transformer's unit of work. Text is split into tokens by a tokenizer, and every token passes through every layer of the network. More tokens mean proportionally more computation and more memory, which is why vendors meter usage in tokens rather than in requests or characters — and why the same text can cost different amounts on models with different tokenizers. See token for how text maps to tokens.
Why output costs more than input
A transformer handles the two directions of a request differently. The prompt is known in full, so all of its tokens can be pushed through the network together in one parallel pass. The answer is not: the model is autoregressive, producing one token, appending it, and running again for the next, with each step reading back stored state for everything that came before. Generation is therefore sequential and keeps expensive hardware busy for longer per token.
That is the usual explanation for why output rates sit above input rates across the market, as the panel below shows from our index. Vendors do not publish their cost models, so treat it as the reason the pattern exists, not as a formula for any one price.
In our indexOutput vs input rates in our index
Counted at build from our price index: the output rate divided by the input rate, for the 163 models with both above zero, among the 167 current public models we price, from 33 vendors. Each rate comes from the vendor's own price page, read between 2026-07-11 and 2026-10-09. How we collect prices.
Why long context is limited, and sometimes dearer
Attention compares every token with every other token, so its work in each layer grows with the square of the sequence length — the paper's own analysis says as much. Doubling a prompt more than doubles that part of the computation, and the stored state the model keeps for earlier tokens grows with every token added. Those two facts are why every model has a maximum context window, and why some vendors charge a higher rate once a prompt passes a threshold; see long-context pricing.
The same stored state is what makes prompt caching possible: if the beginning of a prompt is identical to one processed recently, the work for that part can be reused instead of repeated, and vendors pass part of the saving on as a lower cached-input rate.
Variants you will meet
- Decoder-only — the shape of nearly all chat and coding models: read the prompt, generate the answer token by token.
- Encoder-only — reads text and produces a representation rather than new text; this is the family behind many embedding models.
- Encoder-decoder — the original 2017 layout, one half reading the source and the other writing the output, still used for translation-style tasks.
- Mixture of experts — a transformer in which only some of the network's sub-blocks run for each token, so a very large model costs less to run per token than its size suggests.