Free lesson · AI Engineering
Agentic RAG
Agentic RAG
When the Pipeline Needs to Think
Every RAG system we have built so far follows a fixed pipeline: the query arrives, retrieval runs, context is assembled, the LLM generates. The pipeline is the same for every query - the same number of retrieval calls, the same chunk count, the same context assembly strategy. This works beautifully for straightforward factual questions. It fails on queries that need the system to decide what to do.
Here are three queries that break a fixed pipeline:
"Compare our PostgreSQL and MongoDB configurations for latency-sensitive workloads." This requires two separate retrievals - one for PostgreSQL configuration, one for MongoDB configuration - then synthesis across both result sets. A fixed pipeline retrieves once and gets either PostgreSQL or MongoDB, rarely both at sufficient depth.
"I'm getting a 503 error on the /users endpoint. Help me debug." This query needs iterative retrieval: first search for 503 error documentation, read the result, then search for the specific endpoint's configuration, then possibly search for the load balancer settings referenced in the first result. The diagnosis unfolds over multiple retrieval steps that depend on what earlier steps found.
"Give me a complete overview of our authentication architecture." This is a synthesis query that needs broad retrieval across many documents - auth service docs, SSO integration docs, token management docs, OAuth configuration docs - and the system must decide when it has gathered enough to produce a comprehensive answer.
A fixed pipeline cannot handle these because the retrieval strategy depends on the query. The query determines how many retrievals to run, what to search for at each step, and when to stop. That decision-making capability is what makes RAG "agentic."
What Agentic RAG Is
Agentic RAG gives the LLM control over the retrieval process. Instead of following a fixed pipeline, the model decides:
- Whether to retrieve at all
- What query to use for each retrieval
- Whether the retrieved results are sufficient or whether more retrieval is needed
- When to stop retrieving and generate the final answer
The model becomes a reasoning agent whose primary tool is the retrieval system. The pipeline is no longer fixed - it is a loop where the model plans, retrieves, evaluates, and decides whether to continue.
graph TD
A[User Query] --> B[Agent: Analyse query<br/>What information do I need?]
B --> C[Agent: Plan retrieval<br/>What to search for first?]
C --> D[Execute retrieval<br/>Search vector DB]
D --> E[Agent: Evaluate results<br/>Do I have enough?]
E -->|"Need more<br/>information"| F[Agent: Refine query<br/>What else do I need?]
F --> D
E -->|"Have enough"| G[Agent: Generate answer<br/>from all retrieved context]
G --> H[Response with citations]
E -->|"Results are<br/>irrelevant"| I[Agent: Try different<br/>search strategy]
I --> D
This diagram shows the agentic RAG loop. The user query enters the agent, which analyses what information is needed and plans the first retrieval. After executing the retrieval against the vector database, the agent evaluates the results. If more information is needed, the agent refines the query and retrieves again. If the results are irrelevant, the agent tries a different search strategy. Once the agent determines it has enough information, it generates the final answer with citations from all retrieved context.
The Three Agentic RAG Patterns
Not all agentic RAG is the same. Three patterns represent increasing levels of agent autonomy.
Pattern 1: Query Routing
The lightest agentic pattern. The agent analyses the query and routes it to the appropriate retrieval strategy - not a single strategy for all queries, but a choice among predefined strategies.
ROUTING_PROMPT = """Classify this query into one of these retrieval strategies:
SINGLE: The query asks about one specific topic. Use standard retrieval.
COMPARE: The query asks to compare two or more things. Run separate
retrievals for each thing, then synthesise.
BROAD: The query asks for a comprehensive overview. Run multiple retrievals
with different query facets.
NONE: The query can be answered from conversation context without new retrieval.
Query: {query}
Strategy:"""
async def route_query(query: str) -> str:
strategy = await call_llm(ROUTING_PROMPT.format(query=query),
model="gpt-4o-mini", temperature=0)
return strategy.strip()
# Then dispatch:
if strategy == "SINGLE":
chunks = await single_retrieval(query)
elif strategy == "COMPARE":
facets = await extract_comparison_facets(query) # "PostgreSQL" and "MongoDB"
chunks = await parallel_retrieval(facets) # separate search per facet
elif strategy == "BROAD":
sub_queries = await decompose_query(query) # multiple search angles
chunks = await parallel_retrieval(sub_queries)
Cost: one extra LLM call for routing (~$0.00002 at mini pricing). Minimal latency impact. The comparison and broad strategies add retrieval calls (2–5x) but produce dramatically better results on the query types they handle.
When to use: when your evaluation data shows that query types have different retrieval needs and a single strategy underserves some of them. This is the most practical agentic pattern - most production RAG systems benefit from it.
Pattern 2: Iterative Retrieval
The agent retrieves, reads the results, and decides whether more information is needed - repeating the cycle until satisfied. This handles queries where the information need unfolds as results arrive: the first retrieval reveals that the answer depends on a configuration setting, so the agent searches for that setting next.
async def iterative_rag(query: str, max_iterations: int = 3) -> str:
all_chunks = []
current_query = query
for iteration in range(max_iterations):
# Retrieve
chunks = await retrieve(current_query, top_k=3)
all_chunks.extend(chunks)
# Evaluate: does the agent have enough to answer?
evaluation = await call_llm(
f"""Given the user's question and the information retrieved so far,
decide: can you answer the question completely?
Question: {query}
Retrieved information: {format_chunks(all_chunks)}
If YES, respond with ANSWER.
If NO, respond with SEARCH: [what to search for next]""",
model="gpt-4o-mini", temperature=0,
)
if evaluation.startswith("ANSWER"):
break
elif evaluation.startswith("SEARCH:"):
current_query = evaluation.replace("SEARCH:", "").strip()
# Generate final answer from all accumulated context
return await generate_rag_response(query, all_chunks)
Cost: 2–4 LLM calls per query (routing + evaluation per iteration) plus 2–4 retrieval operations. At mini pricing, the LLM overhead is ~$0.0001 per query. The retrieval operations add 100–300ms per iteration.
When to use: debugging and troubleshooting queries (where the diagnosis unfolds), research queries (where breadth matters), and any scenario where the depth of information needed is unpredictable from the query alone.
Pattern 3: Self-Reflective RAG
The most autonomous pattern. After generating an answer, the agent evaluates its own answer against the retrieved evidence and decides whether the answer is well-grounded or needs improvement.
async def self_reflective_rag(query: str) -> str:
chunks = await retrieve(query, top_k=5)
answer = await generate_rag_response(query, chunks)
# Self-evaluation
reflection = await call_llm(
f"""Evaluate this answer against the source documents.
Question: {query}
Answer: {answer}
Sources: {format_chunks(chunks)}
Check:
1. Does every claim have support in the sources?
2. Does the answer fully address the question?
3. Are there contradictions between the answer and the sources?
If the answer is well-grounded and complete, respond: ACCEPT
If not, respond: REVISE: [what needs to change and what to search for]""",
model="gpt-4o-mini", temperature=0,
)
if reflection.startswith("ACCEPT"):
return answer
else:
# Revise: additional retrieval + regeneration
revision_query = reflection.replace("REVISE:", "").strip()
additional_chunks = await retrieve(revision_query, top_k=3)
return await generate_rag_response(query, chunks + additional_chunks)
Cost: 2 extra LLM calls (generation + reflection) plus potentially one more retrieval and regeneration. Roughly 2x the cost of non-reflective RAG. In exchange, answer quality - particularly faithfulness to sources - improves measurably. In my testing, self-reflective RAG reduced hallucination rates by 30–40% compared to single-pass generation.
When to use: high-stakes applications where answer accuracy justifies the additional cost and latency. Medical information, legal research, financial analysis - domains where a wrong answer has consequences.
sequenceDiagram
participant U as User
participant R as Router
participant S as Search
participant G as Generator
participant E as Evaluator
U->>R: "Compare PostgreSQL and MongoDB<br/>for latency-sensitive workloads"
R->>R: Classify: COMPARE strategy
R->>S: Search "PostgreSQL latency configuration"
R->>S: Search "MongoDB latency configuration"
S->>G: Chunks from both searches
G->>G: Generate comparison answer
G->>E: Answer + source chunks
E->>E: Self-evaluate: are both sides<br/>equally well-supported?
E->>G: ACCEPT - well-grounded comparison
G->>U: Comparison with citations from<br/>both PostgreSQL and MongoDB docs
This sequence diagram shows all three agentic patterns working together on a comparison query. The router classifies the query as a COMPARE strategy. Two parallel searches are executed for PostgreSQL and MongoDB latency configuration. The results feed the generator, which produces a comparison answer. The evaluator self-reflects on whether both sides are equally well-supported. If accepted, the response is delivered with citations from both document sets.
Guardrails for Agentic RAG
Agentic RAG gives the model more autonomy, which means more surface area for failure. Three guardrails are essential:
Iteration limits. Every loop must have a maximum iteration count. An agent that decides "I need more information" indefinitely will run up the token bill without converging. My default: 3 iterations for iterative retrieval, 1 reflection cycle for self-reflective RAG. If the agent has not converged by then, generate the best answer from what has been retrieved and note the gaps.
Cost budgets. Each agentic query should have a maximum token or cost budget. When the cumulative cost of a single query exceeds the budget, stop and generate from available context. Without this, a single complex query can consume 10–20x the cost of a normal query - which is fine occasionally but disastrous if every query triggers the agentic path.
Fallback to fixed pipeline. If the agent's routing, evaluation, or reflection calls fail (LLM errors, timeouts), fall back to the standard non-agentic RAG pipeline. The fixed pipeline is your safety net - it always produces an answer, even if the answer is less optimal than the agentic path would have produced.
Measuring Agent Decision Quality
Agentic RAG introduces a new dimension of evaluation that standard RAG does not have: the quality of the agent's decisions, not just the quality of the final answer.
Three decision-quality metrics worth tracking:
Routing accuracy. Of the queries the agent classified as COMPARE, how many actually required comparison retrieval? Of the queries classified as SINGLE, how many would have benefited from multi-retrieval? Measure by manually reviewing a sample of routed queries weekly. If routing accuracy is below 80%, the routing prompt needs refinement - the agent is either over-routing (sending simple queries to expensive agentic paths) or under-routing (missing complex queries that would benefit from agency).
Iteration efficiency. When the agent decides "I need more information," does the additional retrieval actually improve the answer? Compare the answer quality at each iteration: if iteration 2 rarely improves on iteration 1, the iteration loop is adding cost without value, and you should tighten the convergence criteria.
Reflection calibration. When the self-reflective evaluator rejects an answer, is the rejection justified? A well-calibrated evaluator rejects answers that genuinely have unsupported claims and accepts answers that are well-grounded. An over-calibrated evaluator rejects good answers (wasting a regeneration cycle); an under-calibrated one accepts bad answers (defeating the purpose). Measure by manually scoring a sample of accepted and rejected answers.
These metrics are specific to agentic RAG and supplement - not replace - the standard RAG evaluation from topic 6.9. The agent's decisions are themselves a layer of your quality chain, and they need their own evaluation loop.
When NOT to Use Agentic RAG
The most important section in this topic. Agentic RAG is the most powerful pattern in this module and the most expensive. The temptation to make every RAG pipeline agentic is strong - and wrong.
Do not use agentic RAG for simple factual queries. "What is the rate limit for the API?" does not need query routing, iterative retrieval, or self-reflection. A single retrieval and a single generation answer it correctly. Adding agency to this query type adds 2–5x cost and 500ms–2s latency for zero quality improvement. That is not engineering - it is waste.
Do not use agentic RAG when your fixed pipeline already meets the quality bar. If your evaluation set shows 90%+ accuracy on the standard pipeline, the additional accuracy from agentic patterns may not justify the cost. The marginal improvement matters in high-stakes domains (medical, legal) and matters less in internal tooling and customer support.
Do not use agentic RAG before you have fixed your retrieval fundamentals. An agentic loop on top of bad chunking, a weak embedding model, or no re-ranking is putting a sophisticated steering system on a car with flat tires. Fix the fundamentals (topics 5.1–5.5, 6.2–6.3) first; add agency when the fundamentals are solid and the remaining quality gap is in retrieval strategy, not retrieval execution.
The production pattern I recommend: route to agentic paths selectively. The query router from Pattern 1 classifies each query. Simple factual queries take the fast, cheap fixed pipeline. Comparison, synthesis, and diagnostic queries take the agentic path. You pay the agentic premium only on the 15–30% of queries that benefit from it.
Monitoring Agentic RAG in Production
Agentic pipelines have more moving parts than fixed ones, and each part can fail or degrade independently. Three metrics specific to agentic RAG monitoring:
Iterations per query distribution. Track how many retrieval cycles each agentic query requires. If the median is 1 (most queries converge after one retrieval), your routing is working well - it is sending simple queries to the fixed pipeline and only complex queries to the agentic path. If the median is 3 (hitting the iteration limit frequently), either your retrieval is underperforming (agents cannot find what they need) or your convergence criteria are too strict (agents never decide they have enough).
Agent override rate. How often does the self-reflective evaluator reject the first answer? A 10–20% rejection rate suggests the reflection is catching genuine issues. A 50%+ rejection rate suggests the generation prompt is weak (producing low-quality first answers that require constant correction) - fix the prompt, not the reflection loop.
Cost-per-query distribution. Agentic queries have variable cost (unlike fixed-pipeline queries with near-constant cost). Track the P50, P95, and P99 of per-query cost. Alert on P99 spikes - a single runaway query with 5 iterations and 3 reflections can cost 15x a normal query, and a burst of those constitutes a cost incident.
Interview: The agentic RAG interview question tests cost-quality judgement: "When would you use agentic RAG instead of standard RAG?" The strong answer names the specific query types that benefit (comparison, iterative diagnosis, synthesis), estimates the cost multiplier (2–5x per query), identifies the guardrails needed (iteration limits, cost budgets, fallback), and - crucially - describes the selective routing pattern that applies agency only where it is needed. Candidates who propose agentic RAG for all queries reveal that they have not considered the cost implications.
Exercise: Add Agency to Your RAG Pipeline
Exercise: Implement progressive levels of agency:
- Implement query routing (Pattern 1): classify queries as SINGLE, COMPARE, or BROAD. Test with one query of each type. Does the routed strategy produce better results than the fixed pipeline?
- Implement iterative retrieval (Pattern 2) with a 3-iteration limit: test with a debugging query ("I'm getting error X when Y"). Does the iterative approach find information the single-pass approach misses?
- Implement self-reflective RAG (Pattern 3): after generation, evaluate the answer against sources. Test with a query where the first answer has an unsupported claim. Does the reflection catch it?
- Add cost tracking: log the total tokens and cost per agentic query. Compare to non-agentic queries. What is the cost multiplier for each pattern?
- Challenge: combine all three patterns - route first, iterate if needed, reflect on the answer. Run your full eval set. Does the combined agentic pipeline outperform the fixed pipeline? By how much, and at what cost?
What You Should Take From This Lesson
Agentic RAG replaces the fixed retrieve-augment-generate pipeline with a reasoning loop where the model decides how and when to retrieve. Three patterns offer increasing autonomy: query routing selects the appropriate strategy per query type, iterative retrieval unfolds the information need across multiple search cycles, and self-reflective RAG evaluates its own answer for groundedness. Each pattern improves quality on the query types it targets and costs 2–5x more per query than standard RAG. Guardrails - iteration limits, cost budgets, and fallback to the fixed pipeline - prevent the agent from running unbounded. Use agentic RAG for the query types that demonstrably benefit from it, not as a default upgrade.
One topic remains in this module: evaluating everything we have built. Topic 6.9 introduces the metrics, frameworks, and practices for measuring RAG quality systematically - retrieval precision, generation faithfulness, and end-to-end answer correctness - so you know whether your pipeline is working and where it is not.