Free lesson · AI Engineering

What Is RAG?

Easy11 min readaillmrag

What Is RAG?

The Question That Broke Our First AI Feature

The first AI-powered feature my team shipped at Meta was an internal FAQ assistant. Engineers could ask questions about our infrastructure - "What is the maximum payload size for our internal RPC framework?" or "How do I configure read replicas for our sharded MySQL setup?" - and the assistant would answer.

The model we used was capable. It answered most general software engineering questions correctly. But it could not answer questions about our infrastructure, because our infrastructure documentation existed behind an internal firewall that no training dataset had ever touched. The model had never seen a single page of our internal wiki. When asked about our specific configuration limits, it did one of two things: it either admitted it did not know (fine, but useless), or - more often and more dangerously - it fabricated a plausible-sounding answer that blended public knowledge about similar systems with invented specifics (not fine, and actively harmful when an engineer acted on it).

We needed the model to answer from our documents, not from its training data. But we did not want to fine-tune - the documents changed weekly, and retraining to keep up was operationally impractical (exactly the "fine-tune for form, retrieve for facts" lesson from topic 2.5). We needed the model's generation capability married to our specific, current, authoritative documentation.

That marriage is Retrieval-Augmented Generation - RAG. The idea is simple and powerful: before the model generates an answer, retrieve the relevant documents from your knowledge base and put them in the prompt. The model generates its answer grounded in documents it has just read, not facts it vaguely recalls from training. Our FAQ assistant went from 52% accuracy (hallucination-riddled) to 87% accuracy (grounded in real documentation) in the same week we added the retrieval layer, without changing the model or the generation prompt.

RAG is the most common production pattern in AI Engineering, the most frequently tested in AI Engineering interviews, and the architectural backbone of every knowledge-grounded AI application. This module builds it from scratch, evaluates it rigorously, and extends it into advanced patterns. This first topic gives you the concept, the architecture, and the decision framework for when RAG is the right approach.

The RAG Architecture Pattern

RAG has three stages, each of which maps to infrastructure you have already built:

graph LR
    A[User Query] --> B[1. RETRIEVE<br/>Search your knowledge base<br/>for relevant documents]
    B --> C[2. AUGMENT<br/>Insert retrieved documents<br/>into the LLM's context]
    C --> D[3. GENERATE<br/>LLM produces an answer<br/>grounded in retrieved content]
    
    B -.->|"Module 5:<br/>embeddings +<br/>vector search"| B
    C -.->|"Topic 4.4:<br/>context<br/>engineering"| C
    D -.->|"Module 1:<br/>LLM API call +<br/>structured output"| D

This diagram shows the three stages of Retrieval-Augmented Generation. Stage 1, Retrieve, searches the knowledge base for relevant documents using the embeddings and vector search infrastructure from Module 5. Stage 2, Augment, inserts the retrieved documents into the LLM's context using the context engineering principles from topic 4.4. Stage 3, Generate, produces an answer grounded in the retrieved content using the LLM API call and structured output patterns from Module 1. Each stage maps to previously-built infrastructure.

Stage 1 - Retrieve. The user's query is embedded using the same model that embedded your document corpus (topic 5.2). The embedding is compared against your vector database (topic 5.3) to find the most similar document chunks. The search engine you built in topic 5.4 handles this stage entirely.

Stage 2 - Augment. The retrieved chunks are inserted into the LLM's context - the prompt that will generate the answer. This is context engineering (topic 4.4): selecting which chunks to include, ordering them for attention effectiveness, formatting them with delimiters and metadata annotations, and sizing the total context to stay within budget.

Stage 3 - Generate. The LLM receives the augmented prompt - system instructions plus retrieved context plus the user's question - and generates a response. The generation uses every technique from Modules 1 and 4: appropriate temperature (low, since the answer should be grounded), structured output if needed, streaming for user-facing applications, and the response validation patterns from topic 1.3.

The power of RAG is that it gives the model current, specific, authoritative information at inference time - information that exists in the context (strong attention, reliable recall from topic 2.3) rather than in the model's parameters (compressed, blurry, potentially outdated from topic 2.1). The model's job shifts from recalling facts (unreliable) to synthesising and presenting facts it has just been given (reliable - the strongest capability from topic 2.1's strengths list).

Why LLMs Need RAG

Three fundamental limitations of LLMs make RAG necessary. You have encountered all three in Module 2, but seeing them assembled as the justification for an architectural pattern is a different kind of understanding.

Limitation 1: The Knowledge Cutoff

LLM training data has a fixed endpoint. Everything after that date is invisible to the model. Your internal documentation, published last Tuesday, does not exist in any model's parameters. Neither does last quarter's earnings report, yesterday's API changelog, or the support ticket your user filed this morning. RAG bridges the cutoff by retrieving current information at query time.

Limitation 2: Hallucination on Specifics

From topic 2.1: the model encodes facts as compressed statistical regularities, not as stored records. Well-attested facts (capital of France) are encoded strongly. Specific facts (your company's API rate limit, the exact version of a library that introduced a particular bug) are encoded weakly or not at all - and the model will generate plausible substitutes rather than refuse to answer. RAG replaces parametric recall with retrieved documents, eliminating the hallucination pathway for any fact that exists in your knowledge base.

Limitation 3: No Access to Private Data

No training dataset contains your company's internal wiki, your customer database, your proprietary research, or your product specifications. Fine-tuning on that data is theoretically possible but operationally expensive and fundamentally wrong-tooled (topic 2.5: fine-tune for form, retrieve for facts). RAG gives the model access to private data at inference time, through the context window, without modifying the model's parameters or sending your data into a training pipeline.

Important: RAG is not a workaround - it is the architecturally correct approach. The model's context window was designed for exactly this: processing provided text and reasoning over it. When you give the model a retrieved document and ask it to answer a question, you are using the model at its strongest: transformation of in-context text, not recall from parameters. The context-versus-memory asymmetry from topic 2.1 is not just an observation - it is the design principle that makes RAG work.

RAG vs the Alternatives

RAG is not the only way to give an LLM access to specific knowledge. Understanding the alternatives - and when each wins - prevents you from building RAG where a simpler approach would suffice, or from avoiding RAG where it is clearly the right tool.

Alternative 1: Larger Context Windows

Models with million-token context windows (Gemini 1.5 Pro at 2M tokens) can hold enormous amounts of text directly in the prompt - entire codebases, entire books, entire product catalogs. Why not skip retrieval entirely and just put everything in the context?

When larger context wins: when your knowledge base is small enough to fit (under ~500K tokens), when you need the model to reason across the entire knowledge base simultaneously (cross-document relationships, global summaries), and when the cost of sending the full context on every query is acceptable.

When RAG wins: when your knowledge base is too large for any context window, when most of the knowledge base is irrelevant to any given query (sending everything wastes tokens and dilutes attention), when your knowledge base updates frequently (context-stuffing requires updating every prompt; RAG just updates the index), and when cost matters (the per-query cost of a million-token prompt is orders of magnitude higher than retrieval plus a few thousand tokens of context).

For most production applications, RAG wins - because knowledge bases are large, queries are specific, and budgets are finite.

Alternative 2: Fine-Tuning

From topic 2.5: fine-tuning adjusts the model's parameters to change its behaviour. In principle, you could fine-tune a model on your documentation to teach it your domain knowledge.

When fine-tuning wins: when you need the model to behave differently - respond in a specific format, use domain-specific terminology naturally, follow a particular reasoning pattern. Fine-tuning is excellent for form.

When RAG wins: when you need the model to know specific, current, citable facts. Fine-tuning encodes knowledge as statistical weights that cannot be updated without retraining, cannot be cited back to a source document, and decay in accuracy as the knowledge changes. RAG serves knowledge from documents that can be updated, versioned, and cited.

The engineering mantra: fine-tune for form, RAG for facts. They are complementary, not competing. Some applications use both - a fine-tuned model (for consistent domain-appropriate behaviour) with RAG (for grounded, current facts).

Alternative 3: No External Knowledge

Some tasks do not need knowledge beyond what the model already has. A code reviewer does not need external documents - the code to review is in the prompt. A creative writing assistant does not need a knowledge base. A translation system does not need retrieval. RAG is for knowledge-intensive tasks: question answering, customer support, document analysis, research assistance.

RAG Architecture Variants

The basic RAG pattern - retrieve, augment, generate - has several structural variants that you will encounter in production and in the literature. Understanding them now prevents confusion when someone proposes a "different kind of RAG" that is actually the same pattern with different timing.

Naive RAG is what we have described: retrieve once, generate once. The user asks a question, you retrieve documents, you generate an answer. Simple, effective for single-turn factual queries, and the starting point for everything else.

Iterative RAG runs multiple retrieve-generate cycles. The model generates a preliminary answer, identifies what it is missing, retrieves more targeted documents based on the gap, and generates a refined answer. This is more expensive (multiple LLM calls and retrieval rounds) but handles complex queries that require information from multiple, non-obviously-related documents. Topic 6.8 (Agentic RAG) takes this pattern to its conclusion.

Modular RAG separates the pipeline into independently configurable modules: a query module (reformulation, decomposition), a retrieval module (with swappable strategies from topic 6.3), a context module (compression, re-ranking), and a generation module. Each module can be upgraded, tested, and replaced independently. This is the architectural pattern behind production-grade RAG frameworks like LlamaIndex and Haystack, and it is the architecture your topic 6.2 pipeline will follow.

The variant you choose depends on your quality requirements and latency budget: naive RAG for simple queries at low latency, modular RAG for production flexibility, iterative RAG for complex queries where thoroughness justifies additional time and cost.

The RAG Quality Chain

Here is the most important mental model for this entire module: RAG quality is a chain, and the output quality is limited by the weakest link.

graph TD
    A[Chunking Quality<br/>Are chunks coherent<br/>semantic units?] --> B[Embedding Quality<br/>Do similar queries and<br/>chunks land nearby?]
    B --> C[Retrieval Quality<br/>Are the top-k results<br/>actually relevant?]
    C --> D[Context Engineering<br/>Is retrieved content well-positioned<br/>in the prompt?]
    D --> E[Generation Quality<br/>Does the LLM faithfully<br/>use the context?]
    
    A -.->|"Weak link here =<br/>everything downstream<br/>fails"| E
    
    F[Module 5<br/>topics 5.1-5.5] --> A
    F --> B
    F --> C
    G[Topic 4.4<br/>context engineering] --> D
    H[Module 1<br/>LLM call + prompt] --> E

This diagram shows the RAG quality chain. Five linked stages determine final output quality: chunking quality, embedding quality, retrieval quality, context engineering quality, and generation quality. Each stage depends on the one before it, and a weak link at any stage limits everything downstream. Module 5 feeds the first three stages, topic 4.4 feeds context engineering, and Module 1 feeds generation. A dotted line emphasises that a weak link at the chunking stage causes everything downstream to fail.

Chunking → do your chunks contain coherent, complete units of information? Bad chunks (mid-sentence splits, decontextualised fragments) produce bad embeddings.

Embedding → does your embedding model capture the semantic relationship between queries and relevant chunks? A poorly chosen model (topic 5.2) produces embeddings where relevant content is not close to the query.

Retrieval → given good embeddings, does your search pipeline find the right chunks? Post-processing, filtering, and the scaling techniques from topic 5.5 all affect this.

Context engineering → given relevant chunks, are they positioned and formatted so the LLM attends to them effectively? Topic 4.4's sizing, ordering, and formatting principles apply directly.

Generation → given well-engineered context, does the LLM produce a faithful, grounded answer? Prompt quality, temperature settings, and the model's instruction-following ability all play roles.

The chain model has a practical diagnostic consequence: when RAG answers are wrong, trace backward through the chain to find the weak link. Is the answer wrong because the LLM ignored the context (generation issue)? Because the right context was not in the prompt (context engineering issue)? Because the search returned the wrong chunks (retrieval issue)? Because the relevant content was never chunked properly (chunking issue)? The diagnosis determines the fix, and the fix is almost never "use a better model" - it is almost always upstream.

Interview: The interview version of this topic is universal in AI Engineering loops: "Design a system that answers employee questions about company policies." Every candidate proposes RAG. What separates strong from weak is the depth: strong candidates describe the full quality chain (chunking → embedding → retrieval → context → generation), name specific decisions at each stage (chunk size, embedding model, retrieval method, context ordering, temperature), and identify the evaluation approach (how do you know the system is working?). Weak candidates say "RAG" and jump to the LLM call. The chain is the design; the LLM call is one link.

Exercise: Build Your First RAG Pipeline

Exercise: Assemble the end-to-end pipeline using your Module 5 search engine:

1. Take the semantic search engine from topic 5.4 - it is your retrieval backend.
2. Write a RAG prompt template with three sections: system instructions (role + constraints), retrieved context (with delimiters), and user query.
3. For each query: embed it, search the top 3 chunks, inject them into the prompt, and call the LLM at temperature 0.2.
4. Test with 5 questions whose answers are in your corpus. Score: does the answer faithfully reflect the retrieved content? Does it cite or reference the right source?
5. Test with 2 questions whose answers are NOT in your corpus. Does the model say "I don't have that information" or does it hallucinate?
6. Challenge: ask a question where the retrieval returns a partially relevant document - the right topic but the wrong specific detail. Watch the model extrapolate from partial context. This is the most common RAG failure mode, and you need to see it with your own eyes before topic 6.2 teaches you to prevent it.

What You Should Take From This Lesson

RAG is the pattern of retrieving relevant documents from a knowledge base and placing them in the LLM's context before generation - giving the model current, specific, citable information at inference time instead of relying on parametric recall. It addresses three LLM limitations: knowledge cutoff, hallucination on specifics, and no access to private data. RAG is the right approach for knowledge-intensive tasks with large, updating knowledge bases; larger context windows, fine-tuning, and no-external-knowledge are alternatives for different constraints. Quality flows through a five-link chain - chunking → embedding → retrieval → context → generation - and the weakest link determines the output ceiling. Diagnosing RAG failures means tracing backward through the chain rather than upgrading the model.

The next topic builds the production pipeline: the full end-to-end system from document loading through generation, with the engineering decisions at each stage that separate a tutorial RAG from a production RAG.

Back to AI Engineering