Free lesson · AI Engineering
When to Fine-Tune (and When Not To)
When to Fine-Tune (and When Not To)
The Most Expensive Wrong Decision in AI Engineering
Let me tell you about two teams at Meta that both had the same problem - their LLM output did not match what they needed - and solved it in opposite ways. Both were right. Understanding why is the entire point of this topic.
Team A built a customer support assistant. The responses were accurate but too formal - corporate-speak where the brand voice was casual and friendly. They considered fine-tuning: collect 5,000 examples of ideal responses, train the model to match the tone, deploy the fine-tuned version. Estimated timeline: four weeks of data collection, one week of training and evaluation, ongoing maintenance as the brand voice evolved.
Instead, they spent one afternoon rewriting their system prompt. They added three few-shot examples showing the brand voice, plus a one-sentence instruction: "Respond in a warm, conversational tone - like a knowledgeable friend, not a corporate FAQ." The tone shifted immediately. Zero training data, zero compute, zero deployment complexity. The prompt fix was the right answer because the problem was format - the model already knew how to be casual, it just was not being asked to.
Team B built an internal tool that classified engineering incidents into 47 specific categories from their incident taxonomy. The taxonomy was proprietary - no training data in the world contained these exact 47 categories with their specific boundary definitions. Prompt engineering got accuracy to 76%: good enough to prove the concept, not good enough to replace the manual classification that took engineers 15 minutes per incident. Few-shot examples could not cover 47 categories without blowing the context budget.
They fine-tuned on 2,000 labelled incident-classification pairs. Accuracy rose to 94%. The fine-tuned model had learned the classification boundary between categories - something no amount of in-context instruction could teach at 47-category scale. Fine-tuning was the right answer because the problem was behaviour that could not be described in language - it had to be demonstrated through data.
The decision between these two paths - prompt engineering versus fine-tuning - is the most important technical judgement call in this module, and getting it wrong in either direction is expensive. Unnecessary fine-tuning wastes weeks of data collection and training. Avoiding fine-tuning when it is needed wastes months of diminishing-returns prompt iteration.
The Three Customisation Levers
From Module 2's training pipeline (topic 2.5), you know that an LLM's behaviour comes from three sources: pre-training (raw capability), supervised fine-tuning (instruction following), and alignment (human preferences). The three customisation levers available to you as an AI Engineer map to three different layers of influence over that behaviour.
graph TD
A[Your Application<br/>Needs Custom Behaviour] --> B{What kind of<br/>customisation?}
B -->|"Need specific knowledge<br/>or current facts"| C[RAG<br/>Retrieval-Augmented Generation<br/>Module 6]
B -->|"Need different format,<br/>tone, or instructions"| D[Prompt Engineering<br/>Module 4]
B -->|"Need learned behaviour<br/>that resists description"| E[Fine-Tuning<br/>This module]
C -->|"Adds knowledge<br/>to context window"| F[Cheapest, fastest<br/>Most maintainable]
D -->|"Changes instructions<br/>every call"| G[Cheap, fast<br/>Per-call token cost]
E -->|"Changes model<br/>parameters permanently"| H[Expensive, slow<br/>Highest capability ceiling]
F --> I[Try first]
G --> I
I --> J{Good enough?}
J -->|Yes| K[Ship it]
J -->|No, and the gap<br/>is behavioural| E
This flowchart shows the decision process for choosing a customisation lever. An application needing custom behaviour first determines the type of customisation. Needing specific knowledge leads to RAG, needing different format or tone leads to prompt engineering, and needing learned behaviour that resists verbal description leads to fine-tuning. RAG and prompt engineering are cheaper and faster and should be tried first. Only when both are insufficient and the gap is behavioural should fine-tuning be pursued.
Lever 1: RAG (Retrieval-Augmented Generation). Adds knowledge - current facts, domain-specific information, private data - into the context at query time. Does not change the model's behaviour; changes what it knows during a specific call. Covered in Module 6. The cheapest, fastest, most maintainable lever.
Lever 2: Prompt Engineering. Changes instructions - format, tone, constraints, reasoning approach - sent with every call. Does not change the model's parameters; shapes behaviour through the context. Covered in Module 4. Cheap, fast, iteratable in minutes.
Lever 3: Fine-Tuning. Changes the model itself - adjusts the parameters so the model's default behaviour shifts toward your desired output. Permanent until you retrain. The most powerful lever and the most expensive to operate. This module.
The engineering mantra - stated in topic 2.5 and about to be operationalised: retrieve for facts, prompt for format, fine-tune for behaviour.
What Fine-Tuning Can Change
Fine-tuning modifies the model's parameters based on your training examples. Understanding what kinds of changes stick - and what kinds do not - prevents the two failure modes: fine-tuning for something it cannot do, and skipping fine-tuning for something it excels at.
What Fine-Tuning Is Excellent At
Consistent output format. When your application needs the model to always produce output in a highly specific schema - a 47-field incident classification, a precise XML structure, a domain-specific notation - fine-tuning teaches this through demonstration rather than instruction. The model learns the patterns of the format from hundreds of examples, producing more consistent adherence than even the best-engineered prompt.
Domain-specific tone and style. When your brand voice is specific enough that few-shot examples cannot fully convey it - the vocabulary, the sentence rhythm, the particular way your company frames technical concepts - fine-tuning on thousands of examples in that style shifts the model's default output. The Team A case from the opening could have been fine-tuned; they chose not to because the prompt fix was sufficient. But if the brand voice were more nuanced - say, a medical communication style with specific hedging patterns and regulatory vocabulary - fine-tuning would have been the better tool.
Task-specific classification with many categories. When you have more categories than fit in a prompt's few-shot examples, fine-tuning teaches the decision boundaries between categories through data. This is the Team B case - 47 categories is too many for in-context learning, but 2,000 examples is plenty for fine-tuning.
Reducing unnecessary verbosity or preamble. If your model consistently adds "Sure, I'd be happy to help!" or "Here's the information you requested:" before every response, fine-tuning on examples that go straight to the answer adjusts this tendency at the parameter level - more reliably than prompt instructions, which the model occasionally ignores.
Following complex multi-step procedures. When your task involves a specific sequence of reasoning or analysis steps that the model must follow consistently - a medical triage protocol, a legal document analysis checklist, a code review framework - fine-tuning on examples that demonstrate the procedure teaches the model to follow it by default, without relying on a long instruction set in every prompt.
What Fine-Tuning Cannot Do
Add factual knowledge reliably. This is the lesson from topic 2.5 that this entire module reinforces: fine-tuning does not create a database. It adjusts statistical weights. A fact injected through fine-tuning competes with everything else the parameters encode, degrades unpredictably, and cannot be updated without retraining. The HR team from topic 2.5 - the one that spent four weeks fine-tuning facts - is the canonical example.
Overcome model capability limits. If GPT-4o-mini cannot solve a category of reasoning problem (complex multi-step mathematics, nuanced ethical analysis, creative storytelling at literary quality), fine-tuning GPT-4o-mini on examples of correct solutions will not teach it the underlying capability. It will teach it to produce outputs that look like correct solutions - which is worse than failing, because it fails confidently.
Guarantee correct output. Fine-tuning shifts the probability distribution toward your desired behaviour. It does not guarantee it. A fine-tuned model will still occasionally produce output that deviates from the training examples. Evaluation (topic 9.5) and validation (topic 1.3) remain necessary.
The Decision Framework: A Practical Flowchart
Before starting any fine-tuning project, walk through these five questions. They take 30 minutes and save weeks.
Question 1: Have you exhausted prompt engineering? If you have not tried systematic prompt optimisation (topic 4.7), you are skipping the cheapest lever. Fine-tuning is justified only after prompt engineering has plateaued - "plateaued" meaning you ran the failure-driven optimisation loop and accuracy stopped improving, not "I tried three prompts and it wasn't perfect."
Question 2: Have you tried RAG? If the quality gap is about knowledge (wrong facts, missing context, outdated information), RAG is the answer, not fine-tuning. Fine-tune for form, retrieve for facts.
Question 3: Can you demonstrate the desired behaviour in 5 examples? If yes, few-shot examples in the prompt are almost certainly sufficient (topic 4.3). If the behaviour requires hundreds of examples to learn - because it involves many categories, subtle distinctions, or patterns that resist verbal description - fine-tuning is the tool.
Question 4: Do you have enough quality training data? Fine-tuning requires labelled examples: input-output pairs showing the desired behaviour. How many? Less than you think - 200 to 2,000 high-quality examples is sufficient for most tasks (the LIMA principle from topic 2.5). But they must be high quality: accurate, consistent, representative of the distribution of inputs your model will see. If you do not have the data and cannot create it affordably, fine-tuning is not viable.
Question 5: Can you afford the ongoing maintenance? A fine-tuned model is a deployed model that needs evaluation, monitoring, and occasional retraining as your requirements evolve. The cost is not just the initial training run - it is the lifecycle. If you cannot commit to maintaining the model, prompt engineering (which requires only prompt updates, not retraining) may be the more sustainable choice.
Interview: The fine-tuning interview question is one of the most common in AI Engineering loops: "A team wants to fine-tune a model to improve their RAG system's answer quality. Is this the right approach?" The answer interviewers want: "It depends on what is wrong with the answers." If the problem is wrong facts → fix retrieval, not the model. If the problem is wrong format or tone → try prompt engineering first. If the problem is inconsistent reasoning patterns that resist prompt-level instruction → fine-tuning may be justified, but only with evaluation data showing the gap and quality training data to close it. The decision framework, not the fine-tuning technique, is what they evaluate.
Common Fine-Tuning Misconceptions
Five misconceptions I encounter repeatedly, each one leading to a wasted project:
"Fine-tuning makes the model smarter." It does not. Fine-tuning shifts behaviour, not capability. A model that cannot solve multi-step reasoning before fine-tuning will not solve it after, regardless of your training data. You can teach it to format reasoning steps differently; you cannot teach it reasoning it does not have.
"More data always helps." Covered in depth in topic 9.2, but worth stating here: 10,000 mediocre examples produce worse results than 1,000 excellent ones. The model learns the distribution - including the errors, the inconsistencies, and the filler in your data.
"Fine-tuning is permanent." It is not - in a good way and a bad way. Good: you can fine-tune again from the base model with different data. Bad: fine-tuning can cause catastrophic forgetting - the model becomes excellent at your task but loses competence at general tasks it previously handled well. Topic 9.5 covers detecting this.
"The fine-tuned model no longer needs a system prompt." It does. Fine-tuning shifts the default behaviour, but the system prompt still guides the model on per-request concerns: tone adjustments for different contexts, specific instructions for edge cases, and constraints that were not represented in the training data. Think of fine-tuning as changing the model's defaults and the system prompt as per-call overrides.
"Fine-tuning replaces RAG." The most expensive misconception. Fine-tuning and RAG address different dimensions of the output: fine-tuning shapes how the model responds (behaviour); RAG shapes what it knows for a specific query (knowledge). A production application that needs both domain-specific behaviour and current, attributable facts uses both - and many of the best AI applications I have worked on do exactly that.
The Cost of Fine-Tuning
Let me make the cost explicit, because it shapes the build-versus-buy decision from topic 3.3.
Data collection cost. Creating 1,000 high-quality input-output pairs takes 40–100 hours of domain expert time (manual creation) or $50–500 of LLM-assisted generation (plus human review). The data is the most expensive part, and its quality determines the outcome.
Training cost. On OpenAI's fine-tuning API, training a model on 1,000 examples costs roughly $5–30 depending on the model and the example length. On your own infrastructure, the GPU cost for a LoRA fine-tune of a 7B model is roughly $5–20 per run. Training is cheap; the iteration (typically 3–10 runs before satisfactory quality) multiplies it by the iteration count.
Evaluation cost. Running your evaluation suite (topic 9.5) against each training run, comparing to baseline, and diagnosing failures. Budget 2–5 hours of engineer time per training iteration.
Deployment and maintenance cost. Hosting a fine-tuned model (if self-hosted) or paying the fine-tuned model premium (OpenAI charges 2–10x base model pricing for fine-tuned inference). Plus ongoing monitoring and periodic retraining as requirements drift.
Total cost for a typical fine-tuning project: $500–5,000 in direct costs plus 80–200 hours of engineer and domain expert time over 4–8 weeks. Compare to prompt engineering: $0 in direct costs plus 5–20 hours of engineer time over 1–2 weeks. The cost difference is an order of magnitude, which is why prompt engineering must be exhausted first.
graph LR
A[Prompt Engineering<br/>$0 direct cost<br/>5-20 hours<br/>1-2 weeks] --> B{Good<br/>enough?}
B -->|Yes| C[Ship]
B -->|No| D[RAG<br/>$50-500 setup<br/>20-40 hours<br/>2-4 weeks]
D --> E{Good<br/>enough?}
E -->|Yes| C
E -->|No| F[Fine-Tuning<br/>$500-5,000<br/>80-200 hours<br/>4-8 weeks]
F --> G{Good<br/>enough?}
G -->|Yes| C
G -->|No| H[Larger model<br/>or rethink the task]
This flowchart shows the cost escalation path for customisation. Start with prompt engineering at zero direct cost and 5 to 20 hours over 1 to 2 weeks. If insufficient, move to RAG at 50 to 500 dollars setup cost and 20 to 40 hours over 2 to 4 weeks. If still insufficient, move to fine-tuning at 500 to 5,000 dollars and 80 to 200 hours over 4 to 8 weeks. If fine-tuning is still insufficient, consider a larger model or rethink the task. Each step is an order of magnitude more expensive than the previous.
Exercise: Assess Your Own Fine-Tuning Need
Exercise: Apply the decision framework to a real task:
- Take your best-performing Module 6 RAG pipeline or Module 7 tool-using application. Identify the remaining quality gaps - the cases where the output is not good enough.
- For each gap, classify it: is it a knowledge problem (RAG), a format problem (prompt engineering), or a behaviour problem (fine-tuning candidate)?
- For any behaviour problems, answer the five framework questions. Do you have training data? How much would it cost to create? Can you afford the maintenance?
- Challenge: estimate the total cost of fine-tuning versus continued prompt optimisation for your specific gap. Which is cheaper over a 12-month horizon, accounting for ongoing maintenance?
- Challenge 2: find a task where fine-tuning is clearly NOT the answer - where the problem is knowledge or format, not behaviour. Articulate why to a colleague. This is the most valuable exercise in this topic: knowing when not to fine-tune saves more than knowing how.
What You Should Take From This Lesson
Fine-tuning is the third customisation lever - after RAG (for knowledge) and prompt engineering (for format) - and it changes the model's parameters to shift default behaviour toward your desired output. It excels at consistent output format, domain-specific style, multi-category classification, and complex procedural adherence. It cannot reliably add factual knowledge, overcome model capability limits, or guarantee correct output. The decision framework: exhaust prompt engineering first, then RAG, then fine-tune only when the gap is behavioural, you have quality training data, and you can commit to maintenance. The cost escalation from prompting ($0, days) to fine-tuning ($500–5,000, weeks) is an order of magnitude at each step - which is why each step must demonstrably fail before escalating.
Next: the data. Topic 9.2 covers preparing training data - collection strategies, quality requirements, formatting, and the surprising answer to "how much data do I need?"