What an AI model really costs in production

When a team prices an AI project, they usually start with the model's price list. Then the first production bill arrives and it looks nothing like the estimate. The difference is rarely the price per token. It is how many tokens the system actually uses, and how many other services sit around the model.

Understanding where the money goes is the first step to controlling it.

Where the cost comes from

A single agent request can involve far more than one model call:

• The system prompt and instructions, sent with every call
• Retrieved context, such as documents, product data and conversation history
• Several model calls per task, as the agent plans, calls tools and checks its work
• Retries when a tool fails or an output does not pass validation
• Embeddings for search, generated for every new document and every query
• Infrastructure around the model: vector databases, queues, logging and monitoring

An agent that handles a request in five steps, each carrying a long prompt and several retrieved documents, can cost many times what a one-call estimate suggests. That is the normal shape of agent work, so it needs to be designed for.

Measure cost per task

We track cost per completed task, broken down by step. For a support agent that means cost per resolved ticket. For a content agent it means cost per approved product page.

This measure connects spending to value. If a resolved ticket costs a fraction of what a person's time on that ticket costs, the system is paying off, even if the monthly bill looks large. It also shows which steps are expensive, which is where optimization starts.

Route by difficulty

Many steps run well on a smaller, faster and cheaper model: classifying a request, extracting fields from a form or checking an output against a rule. Complex reasoning and customer-facing writing may need a larger one.

We route each step to the smallest model that passes its part of the evaluation set. The evaluation set is what makes this safe: when we move a step to a cheaper model, we can see right away whether quality held.

Cache what repeats

Many requests share the same instructions, the same policy documents and the same product data. Most providers now offer prompt caching, which makes repeated content much cheaper to send. We structure prompts so the stable parts come first and stay identical between calls.

We also cache at the application level. Answers to common policy questions, product details and search results can be stored and reused for a set time, so the model is not asked the same thing a thousand times a day.

Send less context

Retrieval often returns more than the model needs. Better chunking, reranking and tighter limits on how much history the agent carries reduce the size of every call. This usually improves answer quality as well, because the model has less irrelevant text to sort through.

Batch what can wait

Some work does not need an instant answer: tagging last night's tickets, regenerating product descriptions, summarizing a week of feedback. Batch processing is typically much cheaper than real-time calls and is well suited to these jobs.

Set budgets and alerts

Every production agent we run has a budget and alerts. If cost per task climbs, or total spending moves faster than expected, someone is notified. Often the cause is a bug: an agent stuck in a retry loop or a prompt that grew after a change. Catching it within hours instead of at the end of the month keeps a small problem small.

What we recommend

Price AI by the task, measure cost at every step and use evaluation to move work to cheaper models with confidence. Add caching, trim context and batch what can wait. The aim is a cost per task that clearly beats doing the work by hand, with alerts that catch it if that changes.