What AI actually costs inside your product

Every second product conversation now includes an AI feature. Almost none of them include a number. Founders will argue about ₹8,000 a month of hosting and then wave through a feature whose token bill scales linearly with usage and has no ceiling.

This post is the maths. Not the vendor pricing page — the actual arithmetic we run before we agree to build.

The only formula that matters

Cost per user per month equals calls per user per month, times tokens per call, times price per token.

Everything else is detail. The trap is that founders estimate all three optimistically. They assume five calls when the real number is fifty. They assume a short prompt when the system instructions run to two thousand tokens. They price on the cheap model and then ship the expensive one.

Work an example. A support assistant inside a D2C store:

• 30,000 monthly visitors, 6% open the assistant, so about 1,800 users
• average conversation is 5 turns
• each turn sends roughly 3,000 tokens of context and returns 400

That is 9,000 calls a month, about 27 million input tokens and 3.6 million output tokens. On a frontier model that is a serious monthly bill. On a small model with caching it is pocket change. Same feature, two very different businesses — and the difference is engineering, not luck.

Where the tokens actually go

When we audit an expensive AI feature, the bill is almost never the user’s message. It is the scaffolding around it.

The system prompt. Brand voice, rules, formatting instructions, examples. Often 1,500 to 3,000 tokens, resent on every single call.

Retrieved context. The retrieval step pulls ten documents when three would do. Nobody tuned it after launch.

Conversation history. Turn ten resends turns one through nine. Cost grows steeply across a long conversation unless you summarise older turns.

Retries. A failed parse triggers a retry that costs exactly as much as the original call. Badly specified output formats can double your bill invisibly.

Three of those four are pure engineering choices. Your bill is mostly a design decision, not a market price.

The four levers

1. Route by task

Stop sending everything to the biggest model. Classification, intent detection, tagging, extraction and short rewrites do not need frontier reasoning. Reserve the expensive model for what customers actually see and judge.

A practical split: a small model handles the first pass and resolves 70% of cases, then escalates the rest. Users cannot tell. The invoice can.

2. Cache the unchanging prefix

If your system prompt and reference material do not change between calls, do not pay full price for them repeatedly. Prompt caching charges a reduced rate for a reused prefix. Structure prompts so stable content comes first and variable content last — that ordering is what makes the cache hit.

This is usually the highest-return afternoon of work in the whole feature.

3. Shrink the context

Retrieval quality beats retrieval quantity. Ten mediocre chunks cost more and answer worse than three good ones. Tighten your chunking, add a re-ranking step, and cap what goes into the prompt.

Summarise conversation history after five turns instead of resending it. Cap maximum output length — an unbounded response is an unbounded charge.

4. Do not call the model at all

The cheapest token is the one you never send. Cache complete answers for repeated questions. A support bot answers “where is my order” hundreds of times a day, and that answer comes from your database, not from a language model. Route it with a cheap classifier and answer with real data.

Roughly a third of the traffic in a typical assistant does not need generation at all.

Building the number into your pricing

If you sell a subscription, AI cost is now part of your gross margin. Three rules keep it healthy.

Know your worst user. Averages hide the problem. In every AI product we have shipped, the top 5% of users consume something like 40% of the tokens. Price for them or cap them.

Put the ceiling in the product, not just the contract. Fair-use limits that exist only in terms and conditions do not stop the bill. A counter in your own database does.

Charge for value, meter the abuse. Users accept a generous included allowance plus overage far more easily than a per-message meter on everything. Make the included tier cover 95% of behaviour.

Guardrails we ship on every AI build

• A per-user daily token cap enforced in your code before the call is made
• A provider-level monthly spend limit as the last line of defence
• Every call logged with user id, model, input tokens, output tokens and computed cost
• A daily spend alert to the founder, not just the developer
• A kill switch — one flag that disables the feature without a deploy

That logging table sounds boring. It is the thing that lets you answer “is this feature profitable” with one query instead of a guess.

What we ask before building

Three questions, before any code:

1. What does this feature replace? If it saves a support agent four hours a week, you have a budget. If it exists because AI is on the roadmap, you have a cost centre.
2. What happens if it is wrong? Wrong caption, low stakes. Wrong invoice total, high stakes — and high-stakes features need verification steps that add tokens. Budget for them.
3. What is the cheapest version that tests the hypothesis? Usually a small model, a tight prompt and one screen. Ship that, watch the logs for two weeks, then decide what to spend. That is a proof of concept, not an MVP, and pricing it as one keeps the decision cheap.

The short version

AI features are not expensive. Badly built AI features are expensive, and the gap between the two is caching, routing, context discipline and a cap.

Get the logging in on day one. Everything else you can fix later, but you cannot optimise what you never measured. If the feature is going into a codebase an assistant wrote for you, read the production triage post first — cost problems and correctness problems usually travel together.

Building something with a model in the loop and want the cost modelled before you commit? That conversation is free, and it is usually a spreadsheet rather than a proposal.