Free lesson · AI Engineering

Architecture Patterns for AI Applications

Medium9 min readaillmrag

Architecture Patterns for AI Applications

The Architecture Decision That Determines Everything Else

For ten modules you have built components: LLM calls, prompts, retrieval pipelines, tools, agents, fine-tuned models. Each component works. The question this module answers is: how do you compose them into a system?

The composition matters more than the components. I have seen excellent RAG pipelines inside poorly-architected systems that crashed under load. I have seen mediocre prompts inside well-architected systems that served millions of requests reliably. The architecture - how requests flow, how failures propagate, how scaling works, how data moves - determines whether your AI application is a prototype that works on your laptop or a production system that works at 3 AM when you are asleep.

At Meta, every AI-powered feature goes through an architecture review before deployment. The review does not ask "does the LLM produce good output?" - that is evaluated separately. It asks "what happens when the LLM takes 30 seconds instead of 3? What happens when the embedding service goes down? What happens when traffic spikes 10x? What happens when the LLM returns invalid JSON?" The architecture answers these questions. The components do not.

This module covers the architectural patterns, reliability techniques, data flows, scaling strategies, and operational infrastructure that transform your components into systems. It starts here, with the fundamental choice: which architecture pattern fits your application?

The Four Fundamental Patterns

Every AI application fits one of four architectural patterns - or a hybrid that combines them. Each pattern has distinct strengths, failure modes, cost profiles, and scaling characteristics. Choosing the right one is the first and most consequential architecture decision.

Pattern 1: Request-Response

The simplest pattern: a user sends a request, the system processes it through one or more LLM calls, and returns a response. This is the architecture of every chatbot, every single-turn Q&A system, and every AI-powered API endpoint.

graph LR
    A[Client Request] --> B[API Gateway]
    B --> C[Application Logic<br/>prompt assembly,<br/>parameter selection]
    C --> D[LLM API Call]
    D --> E[Response Processing<br/>validation, formatting,<br/>guardrails]
    E --> F[Client Response]
    
    C -.->|"Optional"| G[RAG Retrieval<br/>Module 6]
    G -.-> C

This diagram shows the request-response architecture pattern. A client request enters through an API gateway, flows to application logic for prompt assembly and parameter selection, makes an LLM API call, passes through response processing for validation, formatting, and guardrails, and returns as a client response. An optional RAG retrieval branch adds knowledge from Module 6's pipeline. The flow is linear and synchronous.

When to use: single-turn interactions (Q&A, classification, extraction, translation), multi-turn conversations where each turn is independent (chatbots where the response is immediate), and any application where the response time must be under 5–10 seconds.

Strengths: simple to build, simple to debug, simple to scale (each request is independent), and predictable latency (one or two LLM calls per request).

Weaknesses: cannot handle tasks that take more than a few seconds, cannot perform multi-step reasoning that spans multiple tool calls, and does not naturally support background processing or async notifications.

Scaling: horizontal - add more API server instances. Each request is stateless and independent. A load balancer distributes requests across instances. This is standard web-service scaling; the LLM call is the only novel component.

Pattern 2: Event-Driven

The system reacts to events rather than (or in addition to) direct requests. A new document is uploaded → the system automatically processes it, generates embeddings, updates the search index. A new support ticket arrives → the system classifies it, routes it, and drafts a response for the agent. A monitoring alert fires → the system investigates automatically.

When to use: applications that respond to external triggers (new data, system events, scheduled tasks), background processing (document ingestion for RAG, batch classification, periodic report generation), and any scenario where the processing does not need to block a user's request.

Strengths: naturally asynchronous (the event producer does not wait for the event consumer to finish), scalable through message queues (add more consumers to handle more events), and decoupled (the event source does not know or care what processes the event).

Weaknesses: harder to debug (events flow through queues and consumers, not through a single request path), eventual consistency (the RAG index is not updated the instant a document is uploaded - there is a processing delay), and more infrastructure (message queues, consumer workers, dead-letter queues for failed events).

Pattern 3: Pipeline

A sequence of processing stages, each transforming the data and passing it to the next. This is Module 7's structured workflow pattern elevated to a system architecture: document ingestion → chunking → embedding → indexing is a pipeline. Query → reformulation → retrieval → re-ranking → generation → citation extraction is a pipeline.

When to use: multi-stage processing where each stage has a clear input and output, data processing workflows (ETL for AI), and applications where the stages can be independently developed, tested, and scaled.

Strengths: each stage is independently testable and deployable, stages can be scaled independently (the embedding stage might need more compute than the chunking stage), and the flow is deterministic and auditable.

Weaknesses: latency is the sum of all stages (a 5-stage pipeline where each stage takes 500ms has 2.5s total latency), a failure in any stage blocks downstream stages, and adding a new stage requires understanding the entire pipeline's data contract.

Pattern 4: Agent-Based

The system uses an autonomous agent (Module 10) as the core processing unit. The agent receives a goal, reasons about what to do, and orchestrates tools, retrieval, and generation dynamically.

When to use: open-ended tasks where the processing path cannot be predicted (investigation, research, complex problem-solving), and applications where the system needs to adapt its approach based on intermediate results.

Strengths: handles complex, multi-step tasks that no fixed pipeline can anticipate, and adapts to novel situations without code changes.

Weaknesses: unpredictable cost and latency (each request may take a different path), harder to test and debug (trajectory evaluation from topic 10.9), and requires the full agent infrastructure from Module 10 (memory, guardrails, human-in-the-loop).

Reference Architectures: Three Common Applications

Rather than leaving the patterns abstract, here are concrete reference architectures for the three most common AI applications.

Reference Architecture 1: Document Q&A System

The most deployed AI application type. A user asks questions; the system answers from a document corpus.

Pattern: Request-response (primary) + Event-driven (ingestion).

Request path: User query → embed query → vector search → retrieve chunks → assemble context → LLM generation → validate response → return answer with citations. Total latency: 1.5–3 seconds. Total LLM calls: 1 (possibly 2 if query reformulation is used).

Ingestion path: New document uploaded → event triggers → extract text → chunk → embed → store in vector DB + document store. Runs asynchronously - the user does not wait. Processing time: 5–30 seconds per document depending on length and format.

What makes it work at scale: the request path is stateless and horizontally scalable. The ingestion path runs on separate workers and does not compete with request processing for resources. The vector database handles the retrieval workload independently from the LLM API.

Reference Architecture 2: AI-Powered Customer Support

A support agent or end-user interacts with an AI assistant that has access to the product knowledge base, customer account data, and ticketing systems.

Pattern: Request-response (chat) + Pipeline (ticket processing) + Event-driven (knowledge base updates).

Chat path: Customer message → classify intent → retrieve relevant docs (RAG) → retrieve customer context (API call) → assemble prompt → LLM generation → validate → respond. The classification step may route some queries to fixed responses (FAQs) without an LLM call - saving cost on the 30–40% of queries that have standard answers.

Ticket processing path: New ticket → classify category → route to team → draft response → queue for agent review. This is a pipeline with LLM steps (classification, drafting) and deterministic steps (routing, queuing).

What makes it work at scale: the FAQ routing bypass reduces LLM load by 30–40%. The ticket processing pipeline runs asynchronously from the chat path. Customer context is cached per session (one API call per session, not per message).

Reference Architecture 3: Internal Investigation Agent

An engineering team uses an AI agent to investigate production incidents - querying metrics, searching logs, checking deployments, and producing diagnostic reports.

Pattern: Agent-based (investigation) + Event-driven (alert trigger).

Investigation path: Alert arrives or engineer requests investigation → agent loop (perceive → reason → act → observe) with access to metrics, logs, deployment history, and documentation tools → agent produces diagnostic report with evidence and recommendations.

What makes it work: the agent is appropriate here because the investigation path genuinely cannot be predicted - different incidents require different tools in different orders. The agent's cost ($0.50–$2.00 per investigation) is justified by the engineering time it saves ($50–

What limits its scale: each investigation consumes 5–15 LLM calls, so the per-investigation cost is 10–30x a simple Q&A interaction. This is sustainable at 20–50 investigations per day but requires cost budgets and monitoring at higher volumes.

Choosing the Right Pattern

graph TD
    A{What kind of<br/>application?} --> B{Immediate<br/>response needed?}
    B -->|"Yes, < 5 seconds"| C{Multi-step<br/>reasoning?}
    C -->|No| D[Request-Response<br/>Simple LLM call + optional RAG]
    C -->|Yes, but<br/>predictable steps| E[Pipeline<br/>Fixed stage sequence]
    C -->|Yes, and<br/>unpredictable path| F[Agent-Based<br/>Dynamic tool orchestration]
    
    B -->|"No, can be<br/>async"| G{Triggered by<br/>external event?}
    G -->|Yes| H[Event-Driven<br/>Queue + consumer workers]
    G -->|No, scheduled| I[Pipeline<br/>Batch processing]
    
    D --> J[Most common pattern<br/>~60% of production AI apps]
    E --> K[Second most common<br/>~25% of production AI apps]
    F --> L[Most powerful, most expensive<br/>~10% of production AI apps]
    H --> M[Background processing<br/>~5% as primary pattern]

This decision flowchart helps choose the right architecture pattern. If an immediate response is needed in under 5 seconds and no multi-step reasoning is required, use request-response - the most common pattern at about 60 percent of production AI apps. If multi-step reasoning is needed with predictable steps, use a pipeline - the second most common at about 25 percent. If the path is unpredictable, use agent-based - the most powerful but most expensive at about 10 percent. If no immediate response is needed and the trigger is an external event, use event-driven for background processing.

The distribution matters: most production AI applications are request-response. The pattern is simple, scalable, and well-understood. The temptation to build an agent-based system when request-response would suffice is a common over-engineering mistake - and the cost difference (one LLM call versus 5–15) makes it an expensive one.

Hybrid Architectures: The Production Reality

Most production systems combine patterns. A typical AI application might use:

Request-response for the user-facing chat interface (immediate response to each message), event-driven for document ingestion (new documents trigger embedding and indexing), pipeline for the RAG retrieval flow (query → retrieval → re-ranking → generation), and agent-based for complex tasks that the user explicitly requests (deep research, multi-step investigation).

The patterns are not mutually exclusive - they are building blocks. The architecture decision is which pattern handles which part of the system, and how the parts connect. A well-architected hybrid routes each request to the simplest pattern that can handle it: a quick factual question goes through request-response, a document upload goes through the event-driven pipeline, and "investigate this production issue" activates the agent.

This is the same "route to the cheapest viable option" principle from Module 3's cost engineering, applied to architecture. Every additional layer of complexity - every agent loop, every pipeline stage, every event consumer - is a cost. Add it only where the simpler pattern demonstrably cannot serve the use case.

Interview: The AI architecture interview question is one of the most common in senior AI Engineering loops: "Design the architecture for an AI-powered customer support system." Strong candidates describe the hybrid: request-response for real-time chat (user asks a question, system responds via RAG), event-driven for ticket processing (new ticket → classify → route → draft response), pipeline for knowledge base updates (new document → chunk → embed → index), and optionally agent-based for complex investigation ("the customer has had this issue three times - investigate the root cause"). Each pattern is justified by the specific requirement it serves. Candidates who propose a single pattern for everything - especially "make it an agent" - reveal that they have not built systems at production scale.

Exercise: Choose Your Architecture

Exercise: Apply the decision framework to three different AI applications:

1. A customer FAQ chatbot that answers questions from a knowledge base. Which pattern? Why? Draw the request flow.
2. A document processing system that extracts key information from uploaded contracts. Which pattern? Why? Draw the data flow.
3. An AI assistant that helps engineers debug production incidents using metrics, logs, and deployment history. Which pattern? Why? Draw the system architecture.
4. For each application: what is the expected latency? What happens when the LLM fails? How does it scale from 10 to 10,000 requests per day?
5. Challenge: design a hybrid architecture that serves all three applications within a single platform. Which components are shared? Where do the patterns intersect?

What You Should Take From This Lesson

Four fundamental architecture patterns - request-response, event-driven, pipeline, and agent-based - serve different application requirements. Request-response dominates (~60% of production AI apps) because it is the simplest, most scalable, and most debuggable. Pipelines handle multi-stage processing with predictable flows (~25%). Agent-based handles complex, unpredictable tasks (~10%). Event-driven handles background processing and external triggers (~5%). Most production systems are hybrids that route each request to the simplest viable pattern. The pattern choice determines cost, latency, scalability, and debuggability - making it the most consequential architecture decision.

Next: what happens when the LLM does not respond. Topic 11.2 covers designing for reliability - the retry strategies, fallback models, circuit breakers, and graceful degradation patterns that keep your system working when its most unpredictable component does not.

Back to AI Engineering