AlgoInsight

Q&A

What is Retrieval-Augmented Generation (RAG)?

By AlgoInsight Team2026-06-18algorithmssystem-design

Overview

LLMs "know" only what was in their training data, and that knowledge has a cutoff date. For applications requiring up-to-date or private knowledge (company docs, recent news, medical records), retraining the LLM is impractical. RAG solves this by turning the LLM into a "reader" that can reference external documents at query time - similar to an open-book exam instead of a closed-book one.

graph LR
    A["User query"] --> B["Embed query<br/>(vector)"]
    B --> C["Vector DB<br/>similarity search"]
    C --> D["Top-k relevant chunks"]
    D --> E["Augmented prompt:<br/>Context: [docs]\nQuestion: [query]"]
    E --> F["LLM generates answer<br/>grounded in retrieved docs"]
    style C fill:#1e1b4b,color:#fff

The RAG Pipeline

Step 1: Indexing (offline)

  1. Chunk documents into smaller pieces (e.g. 512-token chunks with overlap).
  2. Embed each chunk using an embedding model (e.g. text-embedding-3-small, BGE, E5) to produce a dense vector.
  3. Store vectors in a vector database (Pinecone, Weaviate, Qdrant, pgvector, FAISS).

Step 2: Retrieval (at query time)

  1. Embed the user query using the same embedding model.
  2. Search the vector DB for the k most similar chunks by cosine similarity (or dot product for normalized vectors).
  3. Optionally re-rank results with a cross-encoder for higher precision.

Step 3: Generation

Construct a prompt containing the retrieved context and the user query:

System: "Answer using only the provided context. If unsure, say so."
Context:
[chunk 1]
[chunk 2]
[chunk 3]
User: What is the company's refund policy?

The LLM generates an answer grounded in the retrieved documents.

Implementation (Python / LangChain sketch)

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

# Index
vectorstore = Chroma.from_documents(documents, OpenAIEmbeddings())

# Retrieve + Generate
qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4"),
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)
answer = qa.run("What is the return policy?")

Chunking Strategy

Chunking is critical - too small loses context; too large exceeds the context window or dilutes relevance:

Advanced RAG Patterns

Pattern What it adds
HyDE (Hypothetical Document Embedding) Generate a hypothetical answer, embed that, then retrieve - improves recall
Re-ranking Use a cross-encoder to re-score retrieved chunks for precision
Hybrid search Combine dense (vector) + sparse (BM25 keyword) retrieval
Query expansion Generate multiple query variants to improve recall
Self-RAG Model decides when to retrieve and critiques its own output

RAG vs Fine-Tuning

RAG Fine-Tuning
Updates knowledge At query time (dynamic) At training time (static)
Cost to update Re-index documents Retrain / fine-tune the model
Handles private data Yes (in the vector store) Yes (in training data)
Reduces hallucination Yes (grounded citations) Partially
Best for Dynamic, factual Q&A Style, format, specialized tasks

Often the two are combined: fine-tune for style and reasoning, RAG for factual grounding.

Why This Matters in Interviews

RAG is one of the most asked AI system design patterns. Cover: the index-retrieve-generate pipeline, embedding-based similarity search, chunking decisions, the distinction from fine-tuning (dynamic vs static knowledge), and hallucination reduction. Advanced patterns (re-ranking, HyDE, hybrid search) and common failure modes (poor chunking, irrelevant retrieval, context window overflow) distinguish a strong answer.

Common Pitfalls

References

More answers