Open the pricing page of any AI product and you'll see the same confusing units: tokens, credits, or "AI actions." Nobody bills you per word, per request, or per hour of compute directly — and that's not an accident. Understanding what's actually being metered is the difference between a predictable AI budget and a surprise invoice.
What a Token Actually Is
A token is not a word and it's not a character — it's a chunk of text the model was trained to treat as a single unit. Most modern LLMs use a subword tokenizer (byte-pair encoding or a variant of it), which breaks text into frequent fragments rather than whole words.
As a rough example, the word "unbelievable" might split into un, believ, and able — three tokens for one word. Common short words like "the" or "is" are usually a single token. Rare words, made-up names, and non-English text often cost more tokens per word because the tokenizer hasn't seen them often enough to have a dedicated fragment for them.
A useful rule of thumb for English text:
- ~4 characters per token
- ~0.75 words per token
- 1,000 tokens ≈ 750 words ≈ 1.5 pages of text
Code, JSON, and non-Latin scripts (Chinese, Arabic, Hindi) tokenize less efficiently — often 2–4x more tokens for the same amount of information, because the tokenizer's vocabulary was trained mostly on English web text.
import tiktoken
encoding = tiktoken.encoding_for_model("gpt-4o")
tokens = encoding.encode("Understanding tokens saves you money.")
print(len(tokens), tokens)
# 6 tokens — every model call starts with counting these
Why You're Billed on Tokens, Not Requests
Billing per request would be unfair to the provider — a one-line question and a 50-page document analysis cost wildly different amounts of compute. Billing per token ties the price directly to the actual work the model does:
- Input tokens: everything you send — system prompt, conversation history, retrieved documents, the user's message
- Output tokens: everything the model generates back
Almost every provider prices output tokens 3–5x higher than input tokens. Generation is inherently sequential — the model produces one token at a time, each one depending on all the previous ones — while processing input can be parallelized across the whole prompt at once. That asymmetry in compute cost shows up directly in the price.
Why It's Expensive: What You're Actually Paying For
The token price isn't arbitrary — it's a proxy for several real costs stacked on top of each other:
Model size. Larger models (more parameters) require more GPU memory and more compute per token. A frontier model with hundreds of billions of parameters costs meaningfully more per token to run than a smaller distilled version, even though both charge "per token."
Context window cost is non-linear. Transformer attention compares every token to every other token in the context, so compute (and memory) scales roughly with the square of the input length, not linearly. Doubling your context doesn't just double the cost of that call — it can more than double it, which is why long-context requests are disproportionately expensive.
GPU scarcity and utilization. Inference runs on expensive, supply-constrained hardware (H100s, GB200s, TPUs). Providers price in the cost of keeping that hardware running, cooled, and staffed, plus a margin for redundancy and peak load.
Training amortization. Every inference call implicitly pays back a fraction of the tens to hundreds of millions of dollars spent training the model in the first place.
Reliability and safety overhead. Content filtering, rate limiting infrastructure, abuse detection, and redundant serving capacity all add cost that gets folded into the per-token price rather than billed separately.
How Providers Calculate the Price You See
Most providers publish a rate like "$X per 1M input tokens / $Y per 1M output tokens," then apply a few common levers on top:
- Model tiers: a "mini" or "flash" model might be 10–20x cheaper per token than the flagship model, in exchange for lower reasoning quality
- Prompt caching: if you repeatedly send the same system prompt or document context, providers can cache the computed attention state and charge a fraction (often 25–50%) of the normal input rate for the cached portion
- Batch APIs: submitting requests for asynchronous, non-urgent processing (typically ~50% cheaper) since the provider can schedule them into idle GPU capacity
- Credits: a prepaid abstraction layer — 1 credit is simply pre-converted into a fixed number of tokens at a given model's rate, so the provider can normalize pricing across many different underlying models without you needing to think in tokens directly
A simple mental model for estimating a bill:
cost = (input_tokens / 1,000,000 × input_rate)
+ (output_tokens / 1,000,000 × output_rate)
For example, summarizing a 20-page report (~15,000 input tokens) into a 1-page summary (~700 output tokens) on a model priced at $3 / 1M input and $15 / 1M output costs:
(15,000 / 1,000,000 × $3) + (700 / 1,000,000 × $15)
= $0.045 + $0.0105
= ~$0.056 per summary
Run that at 10,000 summaries a month and the bill is ~$560 — trivial per call, significant at scale, which is exactly why token efficiency matters more as usage grows.
How to Actually Use This
Once you understand the unit economics, the cost levers become obvious:
Trim the input, not just the output. System prompts and injected context (RAG chunks, chat history) are usually the largest and most controllable cost driver. Summarize or truncate conversation history instead of resending the full transcript on every turn.
Match the model to the task. Route simple classification, extraction, or short-answer tasks to a small/cheap model, and reserve the flagship model for the subset of requests that genuinely need deep reasoning.
Use prompt caching for repeated context. If every request shares the same long system prompt or document, structure it so the provider can cache and discount that portion instead of re-billing it every call.
Stream and cap output length. Set explicit max-token limits and stop sequences so the model doesn't generate padding or repeat itself — output tokens are the expensive half of the bill.
Monitor at the token level, not the request level. A usage dashboard that only tracks "API calls" hides the real driver of cost. Track input/output tokens per feature so you know exactly which part of the product is expensive, and set budget alerts before a runaway loop or an unbounded context turns into an unexpected invoice.
Tokens aren't a billing gimmick — they're the actual unit of compute the model consumes. Once you can estimate token counts for your own prompts and outputs, AI cost stops being a mystery and becomes just another variable you can engineer around.
If you're building an AI product and want help getting the unit economics right before you scale, get in touch.