The real cost of AI in production

The support agent pilot cost a few dollars a month and convinced everyone. Three months after rolling it out across operations, the model provider's invoice arrives with a number nobody had budgeted for. There was no error: the agent was simply sending each customer's full history, thousands of words, with every query, to answer questions that could often be resolved with a two-line rule.

The scene is a composite and shows a risk in moving from demo to operations. Providers commonly charge for input and output tokens, and some tools carry their own charges. In a pilot, the cost can look small. At scale, tool calls, infrastructure, human review and retries need budgeting too.

The right metric: cost per case resolved

Comparing models' rates per million tokens is useful but incomplete. To calculate cost per resolved case, divide the period's total cost — including failed attempts, retries, tools and human review — by the cases completed correctly. Measure the share left unresolved or handed off separately: low cost means little if many cases are abandoned. A pricier model that gets it right first time can come out cheaper than a budget one that forces a repeat.

Deterministic first

One useful architecture decision is not to use a language model for work that reliable rules can handle. In reconciliation, some matches can be identified by amount, date and reference. A tested rule is usually cheaper and easier to explain for those cases. Ambiguous cases may need a model or human review; the actual proportions depend on process data.

A well-tested rule avoids unnecessary model calls; it still needs maintenance and auditing.

Four ways to control the bill

• Minimal context: send the model only what each task needs, not the full history. The semantic layer helps, because it delivers calculated figures instead of raw tables.
• Prompt caching: a stable prefix may cost less on cache hits. Check minimum length, retention, write cost and actual reuse rate before estimating savings.
• Right-sized models: compare models on your cases and quality thresholds. Use a costlier one only when its improvement justifies the total process cost.
• Batch processing: work that doesn't need an immediate answer can be sent in batches. Some providers offer discounts, subject to their current prices and delivery times.

Measure before you scale

None of this can be optimized without measuring it. An agent in production should report, by case type, how many tokens it uses, how many calls it makes, how long it takes, what it costs and how many cases it resolves, hands off or leaves open. With that data, the conversation stops being “AI is expensive” and becomes concrete: which case type costs most, why, and which part can be resolved with rules.

An agent can cost less or more than the current process. A valid comparison includes quality, supervision, exceptions and maintenance as well as the model's price. Architecture determines how much work goes to the model and how much can be handled more simply.

The scenes in these episodes are composites of common situations, not customer cases. Any figures in them are illustrative.