Costs of AI APIs: tokens explained for finance people
What a token is, why input and output are priced differently, what drives your AI bill, and how to forecast and cap it.
The short answer
AI providers bill per token, a unit nobody outside engineering has met. A token is a fragment of text, roughly three-quarters of a word in English, a little less in other languages. Every call to a model sends tokens in, the instructions, any documents provided, the conversation so far, the question, and receives tokens out, the answer. Providers count both and price them separately, per model, with output priced higher per token. Your monthly bill is the number of calls, times the tokens per call, times the price per token for the model used. Once those three factors are visible in logs, the bill is arithmetic, and it can be forecast, reduced and capped like any other usage cost.
What is in a call
| Component | Direction | Typical size | Notes |
|---|---|---|---|
| System instructions | Input | Hundreds to a few thousand tokens | Sent with every call; cacheable |
| Retrieved documents or context | Input | Hundreds to tens of thousands | The largest and most variable part |
| Conversation history | Input | Grows with each turn | Trim it; do not send everything forever |
| The user’s question | Input | Tens to hundreds | Small |
| The answer | Output | Tens to a few thousand | Priced higher per token; keep concise |
The levers, in order of effect
- Model per step. Small, cheap models for classification, extraction and routine drafting; the strong model only where measured to matter. Often the largest saving.
- Context size. Retrieve passages, not documents. Trim history. Summarise long threads before sending.
- Prompt caching. Providers charge less for repeated identical prefixes such as system instructions; structure prompts to benefit.
- Batching. Non-urgent work sent in batches is often discounted.
- Concise outputs. Ask for the format you need and no more.
- Caps and limits. Provider caps, per-user limits, anomaly alerts. Not a saving on normal days; the difference between an incident and a footnote.
Forecasting for the budget
Take a month of real logs: calls per day, average and maximum input and output tokens, model per step. Multiply by the current price list. That is the baseline. Add expected volume growth. Note which lines are input-dominated, because those respond to context discipline. Set the provider cap at a sensible multiple of the forecast. Revisit quarterly: prices have fallen repeatedly, and a model that was premium last year is often the cheap tier now.
What this means for you
Tokens are just the unit; the bill is calls times size times price. Get the three factors into logs, forecast from a real month, choose models per step, send less context, use caching, and set caps before launch. Then AI usage is a predictable line like bandwidth or postage, reviewed quarterly as prices fall, rather than a surprise on the invoice.
Frequently asked questions
Why is the invoice hard to predict?
Because three variables move: how many calls, how large each call is, and which model handled it. A feature that sends a whole document with every question costs many times one that sends a few passages. A month of real usage logs fixes the first two variables; a decision about models fixes the third. After that the forecast is arithmetic.
Why are output tokens more expensive than input?
Generating text costs the provider more computation than reading it. In practice input still dominates most bills, because prompts, retrieved documents and conversation history are large while answers are short. Reducing what is sent in is usually the bigger saving; keeping answers concise is the smaller one.
How do we stop a runaway bill?
Caps at the provider per day and per month, rate limits per user and per endpoint in the application, alerts on anomalies, and logging of input sizes. Runaway bills are nearly always a loop, a bot or an input size nobody expected, and a cap turns any of them into a small number and an alert.