What is inference in AI?

Training vs inference

A model's life has two very different phases. Training adjusts billions of parameters over enormous datasets until the model is useful; it is done once per model, at the vendor's expense, and the result is a fixed set of weights. Inference uses those fixed weights: every time you send a prompt and get a reply, the model runs once on your input and its weights do not change.

For a buyer of AI, almost all the cost is inference. An API vendor recovers its training spend and serving costs through per-token prices; a company running an open-weight model pays for the GPUs that run inference. Fine-tuning is a small extra round of training on top of an existing model, priced separately where it is offered at all.

What drives inference cost

A language model handles a request in two steps. It first processes the whole input in one parallel pass — often called prefill — then generates the output one token at a time, each step depending on all the tokens before it. That second step occupies the hardware longer per token, which is why APIs price output above input.

Everything else follows from those two steps. Longer prompts cost more prefill; prompt caching lets a provider skip prefill for a repeated prefix and passes part of the saving on. Longer answers and hidden reasoning cost more generation. Bigger models cost more per token than smaller ones, which is why the cheapest and the most expensive current models in our index sit so far apart.

How inference is sold

• Real-time API — the default: per-token rates, answers in seconds.
• Discounted capacity — the batch API (results within 24 hours), flex processing and off-peak pricing trade speed or timing for a lower rate.
• Premium capacity — fast or priority tiers charge more per token for lower latency.
• Self-hosted — you rent or own GPUs and pay for time, busy or idle.

The panel below shows the spread of per-token rates across the current models in our index.