Free lesson · AI Engineering

Prompt Optimisation for Cost and Latency

Hard10 min readaillmrag

Prompt Optimisation for Cost and Latency

The Prompt That Cost

4,000 a Month in Wasted Tokens

At Meta, our AI documentation assistant had a system prompt that grew organically over eight months. Every time a quality issue was discovered, someone added an instruction. "Always cite sources." "Never speculate beyond the provided context." "If the user asks about deprecated APIs, mention the replacement." "Use code blocks for any code longer than one line." "When comparing alternatives, use a table format." "Do not use the word 'simply' as it can feel dismissive."

The system prompt reached 4,200 tokens. Every single request - even a one-sentence question like "What is the rate limit?" - sent 4,200 tokens of system prompt to the model before the user's actual question. At

We rebuilt the system prompt from scratch. Every instruction was tested independently: remove it, run the evaluation dataset, and check whether quality changes. Instructions that had no measurable impact were cut. Instructions that overlapped were merged. The rebuilt prompt was 1,600 tokens - 62% shorter - with identical evaluation scores on every quality metric. Monthly system prompt cost dropped from $33,600 to

This is prompt optimisation: reducing token consumption without reducing output quality. It is the highest-leverage cost optimisation in AI engineering because it reduces cost on every single request, and the savings compound with traffic growth.

Why Token Count Is the Primary Cost Lever

In an LLM-based application, cost is a function of three variables: the number of requests, the average tokens per request (input + output), and the per-token price of the model. You can reduce cost by reducing any of these, but tokens per request is the variable you have the most control over.

Request volume is driven by user demand - reducing it means serving fewer users or degrading the product. Caching reduces effective request volume (Module 11, Topic 11.6) but there is a floor below which cache hit rates cannot improve.

Per-token price is set by the model provider. You can select cheaper models (Module 16, Topic 16.2) but model selection involves quality trade-offs.

Tokens per request is entirely within your control. The system prompt, the retrieval context, the few-shot examples, and the output instructions all contribute tokens that you design. Optimising these is free - it requires no infrastructure changes, no model changes, and no quality compromises if done correctly.

The token budget framework:

For every request, the total token consumption is:

Total tokens = system prompt tokens + retrieved context tokens + user query tokens + few-shot example tokens + output tokens

Each component has an optimisation strategy. The system prompt is optimised through pruning (this topic). Retrieved context is optimised through chunk sizing and retrieval precision (Module 6). Few-shot examples are optimised through example selection and dynamic inclusion. Output tokens are optimised through length constraints and structured output formats.

graph TD
    subgraph Token_Budget["Token Budget Per Request"]
        SP["System Prompt\n1,600 tokens\n(was 4,200)"]
        RC["Retrieved Context\n3,000-6,000 tokens\n(variable)"]
        UQ["User Query\n50-200 tokens\n(not optimisable)"]
        FS["Few-Shot Examples\n0-1,500 tokens\n(dynamic)"]
        OUT["Output\n200-800 tokens\n(constrained)"]
    end
    
    subgraph Optimisation["Optimisation Strategies"]
        O1["Prompt pruning:\nremove redundant instructions"]
        O2["Chunk sizing:\nsmaller, more precise chunks"]
        O3["N/A - user controls this"]
        O4["Dynamic inclusion:\nonly include when needed"]
        O5["Length constraints:\nmax_tokens, format control"]
    end
    
    SP --> O1
    RC --> O2
    UQ --> O3
    FS --> O4
    OUT --> O5
    
    style Token_Budget fill:#1e1b4b,stroke:#818cf8,stroke-width:2px
    style Optimisation fill:#312e81,stroke:#a5b4fc,stroke-width:2px

This diagram shows the five components of the token budget per request and their optimisation strategies. The system prompt at 1,600 tokens (reduced from 4,200) is optimised through prompt pruning. Retrieved context at 3,000 to 6,000 tokens is optimised through chunk sizing for smaller, more precise chunks. The user query at 50 to 200 tokens is not directly optimisable as the user controls it. Few-shot examples at 0 to 1,500 tokens are optimised through dynamic inclusion, only adding them when needed. Output at 200 to 800 tokens is optimised through length constraints and format control.

Prompt Compression Techniques

Technique 1: Instruction Pruning

Remove instructions that do not measurably affect quality. This is the highest-ROI technique because it has zero quality risk when done correctly - if removing an instruction does not change the evaluation results, the instruction was not contributing.

The pruning process:

1. Baseline: run the full evaluation dataset against the current prompt. Record all quality metrics.
2. For each instruction in the system prompt, create a variant with that instruction removed.
3. Run the evaluation dataset against each variant.
4. Any instruction whose removal does not change any quality metric by more than 1 percentage point is a candidate for removal.
5. Remove candidates in groups of 3-5 and re-evaluate to check for interaction effects (two instructions that are individually removable might be jointly necessary).
6. The final pruned prompt passes the full evaluation at quality parity with the original.

Common categories of removable instructions:

Default behaviour reinforcement. Instructions like "Be helpful and accurate" or "Provide clear and concise responses" reinforce behaviour that the model already exhibits by default through its safety training. Removing them rarely changes output quality.

Obsolete patches. Instructions added to fix a specific model behaviour that has since been corrected in newer model versions. "Do not repeat the user's question back to them" was necessary for GPT-3.5 but is unnecessary for GPT-4 and later.

Contradictory pairs. "Provide comprehensive, detailed responses" and "Keep responses concise and to the point" cancel each other out. One must be removed, and the remaining one must be specific about when to apply.

Technique 2: Dynamic Prompt Assembly

Not every request needs every instruction. A system prompt assembled dynamically based on the query type includes only the instructions relevant to that request, reducing token count for queries that do not need the full instruction set.

Implementation:

class DynamicPromptAssembler:
    """Assembles system prompts with only the instructions relevant to each request."""
    
    def __init__(self, instruction_registry: dict[str, str]):
        self.instructions = instruction_registry
        # Map query types to required instructions
        self.type_to_instructions = {
            "factual_question": ["base", "citation", "accuracy"],
            "code_question": ["base", "citation", "code_formatting", "language_detection"],
            "comparison": ["base", "citation", "table_format", "balanced_analysis"],
            "troubleshooting": ["base", "citation", "step_by_step", "code_formatting"],
            "general": ["base", "citation", "accuracy"]
        }
    
    def assemble(self, query: str, query_type: str) -> str:
        """Build a system prompt with only relevant instructions."""
        required = self.type_to_instructions.get(query_type, ["base", "citation"])
        
        prompt_parts = []
        for instruction_key in required:
            if instruction_key in self.instructions:
                prompt_parts.append(self.instructions[instruction_key])
        
        system_prompt = "\n\n".join(prompt_parts)
        return system_prompt

# Example instruction registry
instructions = {
    "base": "You are a technical documentation assistant. Answer questions based on the provided context.",
    "citation": "Cite the specific document and section for every factual claim.",
    "accuracy": "If the provided context does not contain the answer, say 'I don't have information about that in the current documentation.'",
    "code_formatting": "Use fenced code blocks with language tags for all code. Include comments explaining non-obvious logic.",
    "language_detection": "Detect the programming language from context and provide examples in that language.",
    "table_format": "When comparing alternatives, use a markdown table with columns for each comparison dimension.",
    "balanced_analysis": "Present strengths and weaknesses of each alternative. Do not recommend one unless asked.",
    "step_by_step": "Break troubleshooting into numbered steps. Start with the most common cause."
}

Token savings: A factual question uses 3 instructions (~200 tokens) instead of all 8 (~600 tokens). Over 800,000 requests per month, if 60% are factual questions, dynamic assembly saves approximately 192 million tokens per month -

Technique 3: Context Window Optimisation

The retrieved context often contains more text than necessary. Three strategies reduce context tokens without losing relevant information.

Shorter chunks. Reducing chunk size from 1,000 tokens to 500 tokens means each retrieved chunk contains less irrelevant padding around the relevant content. The trade-off is that important context might be split across chunks, requiring more chunks to be retrieved. The optimal chunk size is empirical - test 250, 500, 750, and 1,000 token chunks against your evaluation dataset and measure the quality-vs-token trade-off.

Fewer chunks. Retrieving 3 chunks instead of 5 reduces context by 40%. If the top 3 chunks consistently contain the answer (measurable by checking whether the answer's source appears in the top 3), the additional 2 chunks are adding tokens without adding information.

Context compression. Use a summarisation model to compress retrieved chunks before including them in the prompt. A 500-token chunk compressed to 150 tokens retains the key information while reducing token count by 70%. The cost of the compression model call must be less than the cost of the saved tokens - which is typically the case for applications using expensive models (GPT-4 level), where the compression model (GPT-4o-mini, Claude Haiku) costs 10-20x less per token.

Technique 4: Output Control

Output tokens are typically the most expensive (2-4x the cost of input tokens for most models). Controlling output length is directly controlling the most expensive component.

Max tokens parameter. Set max_tokens to the maximum reasonable output length for each request type. A factual question should not produce a 2,000-token response. Setting max_tokens=300 for factual questions prevents runaway generation without affecting quality - the model produces a concise answer and stops.

Structured output formats. JSON output is typically more concise than prose. If the downstream consumer is code (not a human), use structured output to eliminate narrative padding. A classification response in prose ("Based on my analysis, I believe this query falls into the billing category because...") is 30-50 tokens. In JSON ({"category": "billing", "confidence": 0.94}) it is 12 tokens.

Measuring Prompt Efficiency

Optimisation without measurement is guessing. Three metrics quantify prompt efficiency.

Token efficiency ratio. useful output tokens / total tokens consumed. A request that sends 5,000 input tokens to produce 200 useful output tokens has a 3.8% efficiency ratio. Improving the ratio means more value per dollar spent.

Cost per quality point. monthly inference cost / average quality score. If monthly cost is

Marginal token value. For each component of the prompt (system instructions, context chunks, few-shot examples), measure the quality impact of adding or removing tokens. A system instruction that adds 200 tokens and improves faithfulness by 3 points has a marginal value of 0.015 quality points per token. An instruction that adds 400 tokens with no quality impact has zero marginal value. Cut the zero-value components first.

graph LR
    subgraph Before["Before Optimisation"]
        B1["System prompt: 4,200 tokens"]
        B2["Context: 5,000 tokens"]
        B3["Output: avg 600 tokens"]
        B4["Total: ~9,800 tokens/request"]
        B5["Monthly cost: $33,600"]
    end
    
    subgraph After["After Optimisation"]
        A1["System prompt: 1,600 tokens\n(pruning)"]
        A2["Context: 3,000 tokens\n(chunk sizing)"]
        A3["Output: avg 350 tokens\n(length control)"]
        A4["Total: ~4,950 tokens/request"]
        A5["Monthly cost: 
6,800"]
    end
    
    subgraph Savings["Result"]
        S1["50% token reduction"]
        S2["50% cost reduction"]
        S3["Quality: unchanged\n(verified by eval pipeline)"]
    end
    
    Before --> After
    After --> Savings
    
    style Before fill:#991b1b,stroke:#fca5a5,stroke-width:2px
    style After fill:#166534,stroke:#86efac,stroke-width:2px
    style Savings fill:#312e81,stroke:#a5b4fc,stroke-width:2px

This diagram shows the results of prompt optimisation. Before optimisation: system prompt at 4,200 tokens, context at 5,000 tokens, output averaging 600 tokens, totalling approximately 9,800 tokens per request and 33,600 dollars per month. After optimisation: system prompt reduced to 1,600 tokens through pruning, context reduced to 3,000 tokens through chunk sizing, output reduced to 350 tokens average through length control, totalling approximately 4,950 tokens per request and 16,800 dollars per month. The result is a 50 percent token reduction and 50 percent cost reduction with unchanged quality verified by the evaluation pipeline.

Interview: System design interview connection: "How would you reduce the cost of your AI system?" almost always starts with prompt optimisation because it is the highest-leverage, lowest-risk cost reduction. Interviewers want to see that you understand the token budget framework and can identify which components to optimise first.

What to emphasise: Present the token budget breakdown (system prompt + context + query + few-shot + output). Describe instruction pruning with the evaluation-backed process. Explain dynamic prompt assembly as a technique for reducing per-request token count based on query type. Mention output control (max_tokens, structured output) as a way to reduce the most expensive token category.

What separates strong from weak answers: Weak answers say "make the prompt shorter." Strong answers describe a systematic process: measure the token budget, identify the highest-cost components, test removal against the evaluation dataset, verify quality parity, and track efficiency metrics. The strongest answers provide specific numbers - token counts, cost savings, and the quality metrics that prove the optimisation did not degrade output.

Exercise: Starter: Your system prompt is 2,800 tokens. Design the pruning experiment: how many variants do you create, how do you test each, what quality threshold determines whether an instruction can be removed, and how do you handle interaction effects between instructions?

Intermediate: Your AI feature processes three query types: factual (60% of traffic, needs 2 instructions), analytical (25%, needs 5 instructions), and creative (15%, needs 4 instructions). The full prompt includes all 8 unique instructions at 800 tokens. Design the dynamic prompt assembly system. Calculate the token savings compared to sending all instructions on every request. What is the monthly cost reduction at 500,000 requests/month and

0/million input tokens?

Advanced: Your AI application has a monthly inference budget of $20,000. Current monthly spend is $31,000. The token budget breakdown is: system prompt 2,200 tokens, retrieved context 4,500 tokens (5 chunks × 900 tokens), few-shot examples 1,200 tokens (3 examples × 400 tokens), user query 120 tokens average, output 550 tokens average. Design the optimisation plan to bring spend under $20,000 without reducing quality by more than 2 percentage points. Specify which techniques you apply to each component, the expected token reduction, and the verification process.

Key Takeaways

Prompt optimisation is the highest-leverage cost reduction because it reduces cost on every request without infrastructure changes or model changes. The four techniques - instruction pruning, dynamic prompt assembly, context window optimisation, and output control - each target a different component of the token budget.

Instruction pruning removes instructions that do not measurably affect quality. The pruning process is evaluation-backed: remove an instruction, run the eval dataset, keep the instruction only if its removal degrades quality. At Meta, pruning reduced our system prompt from 4,200 to 1,600 tokens (62% reduction) with identical evaluation scores, saving

Dynamic prompt assembly includes only the instructions relevant to each query type. A factual question does not need table formatting instructions. A code question does not need balanced analysis instructions. The savings are proportional to the diversity of query types and the number of instructions that are type-specific rather than universal.

Measurement is what makes optimisation rigorous rather than speculative. Three metrics quantify prompt efficiency: token efficiency ratio (useful output / total tokens), cost per quality point (spend / quality score), and marginal token value (quality impact per token of each prompt component). Optimise the components with the lowest marginal token value first - they contribute the most cost with the least quality benefit.

Back to AI Engineering