Free lesson · AI Engineering

The True Cost of an LLM Call

Easy11 min readaillmrag

The True Cost of an LLM Call

The Module Nobody Wants to Read First

I placed this module here - after foundations, after understanding LLMs, before prompt engineering, before RAG, before anything else - because I have watched the alternative play out a dozen times and it always ends the same way.

An engineer builds something impressive. A product manager sees it, gets excited, accelerates the roadmap. The feature ships. It works beautifully. Three months later, the monthly cloud bill arrives and contains a line item nobody forecasted, nobody budgeted, and nobody can easily reduce without rearchitecting what is already in production. The feature gets quietly throttled, capped, or in one case I know of personally, turned off entirely - wasting months of engineering time because nobody ran the numbers before building.

This is the module that runs the numbers before building. If you are tempted to skip ahead to the more exciting topics - prompt engineering, RAG, agents - I understand the impulse, and I am asking you to resist it. Every architectural decision in those modules has a cost dimension. An engineer who cannot estimate the monthly bill of a feature they are proposing is an engineer whose proposals get killed in production review.

Let me make the case with one number. The usage field you have been logging since topic 1.1 contains prompt_tokens and completion_tokens. Across this module, I will show you how those two numbers - multiplied by pricing, by traffic, by time - produce monthly costs that range from $50 to $50,000 for architecturally identical features, depending on decisions you have not been taught to think of as cost decisions. Temperature is a cost decision. Schema verbosity is a cost decision. Conversation memory strategy is a cost decision. By the end of this module, you will see cost everywhere, which is the point.

Token Pricing: The Anatomy of the Bill

Every LLM provider charges by the token. The pricing structure is universal, though the rates differ:

Input tokens (prompt_tokens) - what you send. Your system prompt, conversation history, retrieved documents, the user's question - all tokenised and billed at the input rate.

Output tokens (completion_tokens) - what the model generates. The response, from the first token to the stop condition. Billed at the output rate, which is always higher than the input rate.

Why are output tokens more expensive? Topic 2.3 gave you the mechanical answer: input is processed in parallel (prefill, GPU-efficient), output is generated sequentially (decode, GPU-inefficient). Output tokens cost more to produce, so they cost more to buy. The ratio is typically 3–4x.

Here are the numbers as of the time I am writing, for the models you will use most in this course:

Warning: These prices will change. They have dropped by 50–80% across the industry in the last 18 months, and I expect continued decline. The ratios between tiers are more stable than the absolute numbers. Check your provider's current pricing before any cost projection. What will not change is the fundamental structure: input cheaper than output, small models dramatically cheaper than flagships.

Read the right-hand column. The cheapest and most expensive options differ by 100x per token. A feature that costs $200/month on GPT-4o-mini costs $20,000/month on Opus - for identical architecture, identical traffic, identical prompts. The only difference is the model string in your API call. This is why topic 2.6's "start with the cheapest model" is not frugality - it is engineering discipline. Paying 100x for capability you do not need is not a rounding error at production scale.

Anatomy of a Real Call's Cost

Let me walk you through a concrete cost calculation so you can do this reflexively. Take the structured extraction pipeline from topic 1.3 - the invoice parser.

Input composition:

• System prompt with schema and instructions: ~800 tokens
• Invoice text (user message): ~1,200 tokens
• Total input: ~2,000 tokens

Output:

• Structured JSON response: ~300 tokens
• Total output: ~300 tokens

Cost per call on GPT-4o-mini:

Half a thousandth of a cent. Feels free. Now scale it.

At 1,000 invoices/day: $0.48/day →

At 50,000 invoices/day: $24/day → $720/month. Meaningful but manageable.

At 50,000 invoices/day on GPT-4o instead (17x): →

At 50,000 invoices/day on Opus (100x): → $72,000/month. Now you do not have a feature.

Example: The cost of one call is always negligible. The cost of one call × daily volume × 30 days is where budgets live or die. Always compute monthly cost at projected peak volume before choosing your model and architecture. This is not cautious engineering - it is basic due diligence that most tutorials skip entirely.

The Hidden Cost Multipliers

The per-call arithmetic above is the foundation. On top of it sit five multipliers that inflate the real bill beyond what naive per-call math predicts.

Multiplier 1: Conversation History (The Quadratic Tax)

From topic 1.5, you know that multi-turn conversations resend the entire history on every turn, making input costs grow quadratically with conversation length. Let me put concrete numbers on it for the first time, because this multiplier alone has been the largest surprise cost in every chat product I have advised.

A 10-turn conversation with 200-token messages and 400-token responses sends roughly:

• Turn 1: 800 tokens (system) + 200 = 1,000 input tokens
• Turn 5: 800 + 5×(200+400) = 3,800 input tokens
• Turn 10: 800 + 10×(200+400) = 6,800 input tokens
• Cumulative input across 10 turns: ~39,000 tokens

Compare to the naive estimate - "10 turns × ~200 input tokens = 2,000 input tokens total" - and you see a 20x gap. The history resend is invisible in per-call pricing, and invisible to anyone who has not read topic 1.5. But it is the dominant cost driver for any conversational application.

Your topic 1.5 memory strategies - sliding window, summarisation, hybrid - are not just user-experience tools. They are cost tools. A sliding window of 4 turns caps the input growth, cutting cumulative input from ~39,000 to roughly ~20,000 tokens for the same 10-turn conversation. Summarisation compresses further. The conversation memory architecture is a budget architecture.

Multiplier 2: Retries

Topic 1.3's validate-retry loop sends the original prompt plus the failed output plus error feedback, then generates again. Each retry roughly doubles the tokens for that call. A 3% retry rate adds 3% to your bill. A 20% retry rate - a symptom of a bad prompt or over-strict validation - adds 20% or more, because retries tend to involve longer inputs (the accumulated feedback). Monitor retry rate as a cost metric, not just a quality metric.

Multiplier 3: Retrieval Context (The RAG Tax)

RAG applications (Module 6) stuff retrieved documents into the prompt - typically 2,000 to 8,000 additional input tokens per call. Those tokens appear in every request. A RAG system with 4,000 tokens of retrieved context costs 2–3x more per call than the same application without retrieval, even before accounting for the embedding and vector-search costs of the retrieval itself. This is the right trade-off for quality (grounded answers versus hallucinated ones), but it is a trade-off, and it scales with traffic.

Multiplier 4: Agent Loops

Agent applications (Module 10) make multiple LLM calls per user request - one to reason, one to select a tool, one to process the tool's output, one to formulate the response. A simple agent might make 3–5 calls per interaction; a complex one, 10–20. Each call carries its own input and output cost. The bill for an agent interaction is the bill for an entire mini-conversation with the model talking to itself. Agents are the most expensive pattern in AI Engineering, and Module 10 will teach you to budget for them before building them.

Multiplier 5: Peak-to-Average Ratio

Production traffic is not uniform. Support chat volumes spike on Monday mornings. E-commerce extraction spikes during sales events. Your average daily cost is not your peak daily cost. Budget for the peak, monitor for sustained peaks that indicate a change in baseline, and set cost alerts at both levels.

The Cost Reducer You Already Have: Prompt Caching

One lever works in your favour without architectural changes: prompt caching. From topic 2.3, you know that providers can reuse the KV cache for repeated prompt prefixes, and many now pass part of that saving to you - Anthropic offers up to 90% discount on cached input tokens, and OpenAI offers 50%. Because your system prompt and often your conversation prefix are identical across requests, a meaningful fraction of your input tokens may qualify.

The engineering implication is concrete: structure your prompts with stable content first (system prompt, static instructions, few-shot examples), variable content last (retrieved documents, user message). Two prompts with identical capability but different ordering - one cache-friendly, one not - can differ in effective input cost by 2–3x. This is the architectural principle from topic 2.3, now visible as a line item in your cost model. We build on it properly in Module 16, but you can capture the benefit from day one by getting the order right.

The Costs That Are Not on the Pricing Page

Token costs are the line item you see. Three other costs show up in practice and never appear on a provider's pricing page.

Embedding costs. RAG applications (Module 6) convert documents into embeddings before they can be searched. Embedding models charge per token too - typically 10–50x cheaper than generation models, but the volume is different: you embed your entire document corpus upfront and then re-embed on every update. A corpus of 100,000 documents at ~500 tokens each is 50 million tokens of embedding - roughly

Vector database costs. Storing and querying those embeddings requires infrastructure. Managed vector databases (Pinecone, Weaviate Cloud) charge by storage and query volume. Self-hosted options (pgvector, Qdrant, Chroma) cost whatever your server costs. Module 5 covers selection; for cost modelling purposes, estimate $50–500/month for a typical production workload, scaling with corpus size and query rate.

Compute for self-hosted models. If your cost model says "self-host" (a decision Module 3's final topic addresses), you are buying GPU time instead of API tokens. A single A100 GPU - the minimum for running a 70B model - costs roughly

Operational overhead. Monitoring, incident response, prompt iteration, evaluation - all of these consume engineering hours. A feature with a $200/month token bill and a 20-hour/month operational burden is more expensive than it looks. This cost is harder to model and easier to underestimate. I will not pretend to give you a formula for it; I will say that every team I have advised has underestimated it by at least 2x in their first year.

Building the Cost Estimate: A Template

Here is the template I use for every AI feature proposal at Meta, adapted for your use. Fill this in before you write any code, and update it at each architectural decision point.

Exercise: Apply this template to three Module 1 applications:

1. The invoice extractor (single call, structured output). Estimate at 1,000 and 50,000 invoices/day on both mini and flagship models.
2. The streaming chat assistant (multi-turn, 8 average turns). Estimate with and without the sliding-window memory strategy. Quantify what the window saves you per month.
3. A hypothetical customer support bot: 5,000 conversations/day, 12 average turns, with a RAG context of 4,000 tokens. Which model tier can you afford at a $2,000/month budget?
4. Challenge: take your actual Module 1 usage logs and compute your real per-call cost from measured tokens and current pricing. Compare to your naive estimate. The gap - always larger than expected - is this module's entire argument.

The Psychological Trap: It Feels Free

One last thing, because it is the reason so many teams get caught.

LLM APIs have the psychology of free software. You sign up, get a key, make a call, see a response, and the cost - $0.0005 - is genuinely too small to feel. Every individual call is affordable. The prototype is affordable. The demo is affordable. The staging environment running 200 test requests a day is affordable.

The shift happens when production traffic multiplies every number by 10,000x. Suddenly the per-call cost that felt like zero is $5,000/month. But by then the architecture is set, the model is chosen, the prompts are written, and changing any of them is a project.

This is why cost modelling comes before building, not after. Not because I enjoy spreadsheets - though I do, and the founding story of AlgoInsight involves a spreadsheet, so I come by it honestly - but because the numbers are simple, the arithmetic takes thirty minutes, and the alternative is discovering your bill three months after you could have changed it cheaply.

What You Should Take From This Lesson

LLM costs are per-token with output tokens 3–4x more expensive than input tokens, and the tier gap between small and flagship models is ~100x - making model selection the single biggest cost lever. But per-call cost is deceiving: the real bill emerges from five multipliers - conversation history's quadratic growth, retry overhead, RAG context inflation, agent call multiplication, and peak-to-average traffic ratio - each of which can independently double or triple the naive estimate. Build the cost template before writing code, compute at projected peak volume, and treat every architectural decision from this point forward as having a cost column alongside a quality column.

The next topic takes this per-call arithmetic and builds it into a proper cost model - one you can present to leadership, project over 12 months, and use to make the build-versus-buy decision that determines whether your AI feature lives on an API or on your own GPUs.

Back to AI Engineering