Free lesson · AI Engineering
The Transformer Architecture
The Transformer Architecture
Why an Application Engineer Needs to Look Inside
Let me set expectations before we open this box, because "transformer architecture" is where most AI courses lose their audience - either by drowning engineers in matrix calculus they will never use, or by hand-waving so hard that nothing sticks.
Here is my contract with you for this topic. You will not implement a transformer. You will not compute attention weights by hand. You will not need linear algebra beyond "vectors are lists of numbers and similar vectors point in similar directions." What you will get is the intuition that turns four production mysteries you have already encountered into obvious consequences:
- Why input tokens cost 3–4x less than output tokens (topic 1.1's pricing asymmetry)
- Why prompt caching can slash your bill without changing your architecture (topic 1.5's teaser)
- Why context windows have hard limits and why long context is expensive
- Why information "gets lost in the middle" of long conversations (topic 1.5's soft cap)
That is the payoff: not academic completeness, but the mechanical understanding that lets you reason about cost, latency, and quality instead of memorising provider documentation. My own moment of conversion came when a pricing question from a partner team - "why are we billed differently for input and output when it's the same model?" - turned out to have a precise architectural answer. Once you see it, you cannot unsee it, and a dozen API behaviours snap into focus at once.
From Token IDs to Vectors: Embeddings
The last topic ended with your text as a sequence of integer IDs. Integers are labels, not meaning - token 9822 is not "more similar" to token 9823 in any useful sense. The model's first move is to convert each ID into an embedding: a vector of numbers (thousands of dimensions in modern models) that does encode meaning, in a specific sense: tokens that behave similarly in text get vectors that point in similar directions.
These embedding vectors are learned during training, and they are the first place the model's statistical knowledge lives. The vector for " Paris" ends up geometrically near the vectors for " London" and " Berlin", in a region of the space you might informally call "capital-city-ness." The vector for " king" minus " man" plus " woman" famously lands near " queen" - meaning as geometry.
Hold onto this concept firmly, because it escapes the model's internals and becomes a product you use directly: Module 5 is built entirely on embedding vectors as a standalone tool for search and retrieval. For today, the point is simpler - after the embedding step, your prompt is a sequence of meaning-vectors, one per token, and everything that follows is operations on those vectors.
One more ingredient goes in at this stage: positional information. The attention mechanism we are about to meet is, by itself, order-blind - it would process "dog bites man" and "man bites dog" identically. So each token's vector gets stamped with its position in the sequence (modern models use a clever rotation-based scheme called RoPE, but the mechanics are unimportant here). What matters: position is encoded into the math, not free - which is part of why extending a model's context window beyond what it was trained for is a genuine research problem and not a config flag.
Attention: The Mechanism That Is the Whole Show
Now the centrepiece. The problem attention solves is context. The word "bank" means one thing in "river bank" and another in "bank transfer." The token " it" in a long sentence refers to something specific earlier. A static per-token vector cannot capture this - the vector for " bank" is the same integers regardless of neighbours. The model needs each token's representation to be informed by the other tokens around it.
Attention is the mechanism: every token looks at every other token before it, scores each one for relevance, and updates its own vector as a relevance-weighted blend of what it found.
The mechanics, at the intuition level: each token's vector generates three things - a query ("here is what I am looking for"), a key ("here is what I contain"), and a value ("here is what I will contribute if you attend to me"). Token by token, queries are compared against all previous keys; high query-key similarity means high relevance; and each token's new representation is the weighted sum of the values, weighted by those relevance scores. The token " it" emits a query shaped like "seeking a recent noun"; the key for " the invoice" matches strongly; and " it"'s updated vector now contains invoice-ness. Reference resolved - not by rules, but by learned vector geometry.
Two multiplications on this idea give the full architecture:
Multi-head attention. One relevance-scoring scheme is not enough - syntax, coreference, topic, tone are different kinds of relevance. So each layer runs many attention "heads" in parallel (dozens in large models), each with its own learned notion of what to look for. One head might track subject-verb structure; another, long-range topic; another, code bracket matching. Their findings are combined into the token's updated vector.
Stacked layers. The attention step (plus a small per-token neural network called the feed-forward layer, which you can think of as "processing what attention gathered") forms one transformer block. Modern models stack many dozens of these blocks. Early layers resolve local, surface-level relationships; deeper layers compose them into increasingly abstract representations - from "these tokens form a noun phrase" up through "this paragraph is sarcastic" and "this function has an off-by-one bug." By the final layer, the last token's vector has absorbed, through dozens of rounds of look-at-everything-and-blend, a representation of the entire sequence's meaning as it bears on what comes next - and one final projection converts that vector into the probability distribution over 100K tokens that topic 2.1 promised you.
That is the whole architecture. Embed, then attend-and-process, stacked deep, then project to a distribution. Everything else is refinement.
graph TD
A[Token IDs from tokeniser] --> B[Embedding layer<br/>IDs become meaning-vectors<br/>+ positional encoding]
B --> C[Transformer block 1<br/>multi-head attention + feed-forward]
C --> D[Transformer block 2]
D --> E[... dozens more blocks ...]
E --> F[Final token's vector<br/>context-saturated representation]
F --> G[Projection to vocabulary<br/>probability distribution over ~100K tokens]
C -.->|each block: every token attends<br/>to every previous token| C
This diagram shows the transformer architecture as a pipeline. Token IDs from the tokeniser pass through an embedding layer that converts them to meaning-vectors with positional encoding. The vectors then flow through dozens of stacked transformer blocks, each performing multi-head attention followed by a feed-forward network, with every token attending to every previous token in each block. The final token's context-saturated vector is projected onto the vocabulary to produce the probability distribution over roughly one hundred thousand tokens.
Note: If you want one sentence to carry out of this section: attention lets every token's meaning be rewritten by every token before it, and stacking that operation dozens of times deep is what turns statistics into apparent understanding. The 2017 paper that introduced this was titled "Attention Is All You Need," and seven years of frantic research have mostly confirmed the title.
The Four Payoffs: Architecture Explains Your Production Life
Now we collect. Each of the four mysteries from the opening falls out of what you just learned.
Payoff 1: Why Input Tokens Are Cheaper - Prefill vs Decode
When your prompt arrives, the model must compute attention for all its tokens - but here is the crucial fact: all prompt tokens can be processed in parallel, because they all already exist. The GPU chews through your entire 5,000-token prompt in one massively parallel pass. This phase is called prefill, and it is what dominates your time-to-first-token.
Generation is different. Each new token depends on the one before it - you cannot compute token N+1's attention until token N exists. Generation, called decode, is inherently sequential: one full pass through all those layers per token, no parallelism across the output. Sequential GPU work is expensive work.
There is the pricing asymmetry from topic 1.1, mechanically explained: input tokens are batch-parallel and cheap to process; output tokens are one-at-a-time and expensive. The 3–4x price ratio is not a business decision - it is the silicon economics of prefill versus decode showing through the invoice.
Payoff 2: Why Prompt Caching Works - The KV Cache
During prefill, the model computes keys and values for every prompt token - and during decode, every new token attends back over all of them. Recomputing all those keys and values for every generated token would be ruinous, so inference engines store them: the KV cache, the single most important piece of inference infrastructure you have never heard of.
Now the provider-level insight: if two requests share an identical prefix - say, your 2,000-token system prompt, or the stable early history of a conversation - the keys and values for that prefix are identical and can be computed once, stored, and reused across requests. That is prompt caching, the economic lever teased in topic 1.5, demystified: the provider is literally skipping prefill work it has already done, and passing part of the savings to you. It also hands you a design rule you can apply today: structure prompts with stable content first, variable content last. A prompt that puts the user's message before the system instructions destroys its own cacheability; the same content reordered caches beautifully. The caching module builds the full strategy; the architecture is why it works.
Payoff 3: Why Context Is Limited and Long Context Is Expensive
Every token attends to every previous token. For a sequence of N tokens, that is on the order of N² attention computations, and a KV cache whose memory footprint grows with N for every layer and every head. Double your context and you roughly quadruple attention compute while linearly inflating a very large cache that must live in scarce GPU memory.
That quadratic-and-memory wall is why context windows are a hard architectural budget rather than a dial; why million-token contexts required genuine research breakthroughs (clever attention approximations and memory management, each with trade-offs); and why long-context requests are slower and pricier even when they fit. When topic 1.5 told you a conversation has a hard cap, this is the wall the cap is protecting you from.
Payoff 4: Why the Middle Gets Lost
The soft cap, too, has an architectural face. Attention scores are a competition - every token's relevance is weighted against every other token's. In a 60,000-token context, each individual token competes with 59,999 others for attention mass; important details receive fainter weights simply because the denominator grew. Empirically, models attend most reliably to the beginning of context (heavily reinforced during training, where sequence starts carry framing) and the end (recency, and proximity to the token being generated), with a measurable sag in between - the "lost in the middle" effect you met as a warning in topic 1.5, now visible as attention arithmetic.
The engineering translations are immediate: put critical instructions at the start (system prompt) and critical facts near the end (right before the question); do not assume a fact is "in context" means "will be used"; and prefer less, better-placed context over more - a principle that becomes the beating heart of retrieval design in Module 6.
Important: Notice what just happened across these four payoffs: pricing pages, caching discounts, context limits, and quality degradation - four things that looked like arbitrary provider policy from Module 1's black-box view - are all one architecture viewed from different angles. This is why the module exists. You cannot negotiate with N², but you can design for it, and now you know that is what you are doing.
Interview: The interview form of this topic is rarely "explain a transformer." It is: "Why is time-to-first-token high on long prompts but generation speed unchanged?" (prefill is parallel but proportional to input; decode is per-token) - or "How would you structure prompts to maximise cache hits?" (stable prefix first) - or "Why does your RAG system's accuracy drop when you stuff more documents into context?" (attention competition, lost-in-the-middle). Interviewers use these to distinguish engineers who understand the machine from engineers who memorised the pricing page. You now belong to the first group.
Exercise: Observe the Architecture From Outside
Exercise: You cannot open the provider's model, but its architecture casts measurable shadows. Use your Module 1 instrumented client:
- Send prompts of 100, 1,000, 5,000, and 15,000 tokens (pad with repeated text), each requesting a fixed 50-token answer. Plot TTFT against input size, and tokens-per-second of generation against input size. You should see TTFT climb (prefill scales with input) while generation speed stays roughly flat (decode is per-token). You have just measured prefill versus decode.
- If your provider exposes prompt caching, send the same long-system-prompt request twice in quick succession and compare costs and TTFT. Then break the cache by changing one early word, and watch the discount vanish. You have just measured the KV cache.
- Bury a specific fact ("the access code is 7291") at the start, middle, and end of a ~10,000-token context in three separate runs, and query for it at temperature 0, several times each. Score recall by position. You have just measured lost-in-the-middle.
- Challenge: using your position data from step 3, rewrite the "conversation memory" hybrid from topic 1.5 to place the rolling summary and pinned facts in the architecturally strongest positions. Justify the layout in one paragraph using attention competition.
One More Room in the Box: Where the Facts Live
Attention deservedly gets the spotlight, but I owe you one honest paragraph on its quieter partner, because it pays off two modules from now. Each transformer block pairs attention with a feed-forward network - a per-token neural layer that processes what attention gathered - and in large models, the feed-forward layers hold the majority of the parameters. Interpretability research increasingly suggests a division of labour: attention routes information between tokens, while the feed-forward layers act as the model's associative store - the place where topic 2.1's "compressed statistical knowledge" physically lives. When the model completes "The Eiffel Tower is located in" with " Paris", attention assembled the query; the feed-forward layers supplied the association.
Why should an application engineer care where facts live? Two reasons with direct consequences. First, it sharpens the context-versus-memory asymmetry from topic 2.1: facts in your prompt reach the model through attention - precise, inspectable, current - while facts from its parameters come out of a compressed associative store that blurs, conflates, and dates. Same output text, profoundly different reliability provenance. Every time you choose between "hope the model knows it" and "put it in the context," you are choosing between these two pathways, and now you know why the second one wins.
Second, it previews a design decision waiting in Module 9: fine-tuning updates these parameters - which makes it excellent at teaching the model behaviours, formats, and styles (broad statistical patterns, exactly what parameters encode well) and a poor, expensive way to teach it facts (which belong in retrieval, where they stay current and attributable). "Fine-tune for form, retrieve for facts" will be a load-bearing rule in that module, and its foundation is this paragraph: the architecture stores patterns beautifully and facts blurrily, so put each where the machine handles it best.
What You Should Take From This Lesson
The transformer is one pipeline: token IDs become meaning-vectors with position stamped in; dozens of stacked blocks let every token repeatedly attend to - score and absorb - every token before it, through many parallel heads capturing different kinds of relevance; and the final vector projects to the next-token distribution. From that single design flow the four production facts you now own mechanically: prefill is parallel and cheap while decode is sequential and expensive (your pricing asymmetry and your latency profile); the KV cache makes shared prefixes reusable (your prompt caching, and your prompts-stable-content-first design rule); attention's quadratic cost walls in the context window (your hard cap); and attention competition fades the middle of long contexts (your soft cap, and your place-critical-content-at-the-edges rule).
You now know what the machine is. The next topic watches it run: the full generation loop from prefill through sampling to the stop token - where the temperature and top-p dials from topic 1.2 finally connect to the machinery beneath them, and where you will learn precisely why even temperature 0 could not give you determinism.