Q&A

What is the Transformer architecture?

By AlgoInsight Team2026-06-18data-structuresoperating-systemssystem-design

Overview

Before Transformers, sequential models (RNNs, LSTMs) processed text one token at a time - making them slow and poor at capturing long-range dependencies. Transformers process the entire sequence at once via attention, enabling parallelization on GPUs and far better capture of long-range context. This architectural shift unlocked training at scale, which produced modern LLMs.

graph TD
    A[Input tokens] --> B[Token + Positional Embeddings]
    B --> C["Multi-Head Self-Attention<br/>(each token attends to all others)"]
    C --> D[Add & Norm residual]
    D --> E["Feed-Forward Network<br/>(per-position MLP)"]
    E --> F[Add & Norm residual]
    F --> G[Next encoder layer or output]
    style C fill:#1e1b4b,color:#fff

Self-Attention: The Core Mechanism

For each token, self-attention computes three vectors - Query (Q), Key (K), and Value (V) - from learned linear projections of the token's embedding:

Attention(Q, K, V) = softmax( QKᵀ / √dₖ ) × V

Intuitively: the dot product QKᵀ measures how relevant each token (key) is to the current token (query); the softmax turns scores into attention weights; multiplying by V produces a weighted sum of value vectors - a context-aware representation.

Scaling by √dₖ prevents the dot products from growing so large that the softmax saturates to near-zero gradients.

Multi-Head Attention

Rather than one attention computation, multi-head attention runs h attention heads in parallel, each learning different aspects of relationships (syntax, coreference, semantics), then concatenates and projects the results:

MultiHead(Q,K,V) = Concat(head₁, ..., headₕ) × Wᴼ

This lets the model jointly attend to information from different representation subspaces.

Positional Encoding

Attention has no inherent notion of order (it's permutation-equivariant). Positional encodings inject position information - either fixed sinusoidal encodings (original Transformer) or learned positional embeddings (GPT), or RoPE (Rotary Position Embedding, used by LLaMA and many modern LLMs).

Encoder vs Decoder vs Encoder-Decoder

Decoder-only models with causal (masked) attention are now the dominant paradigm - all tokens can only attend to previous tokens, so the model predicts the next token autoregressively.

Feed-Forward Sub-layer

After attention, each position passes independently through a 2-layer MLP (same weights across all positions):

FFN(x) = ReLU(xW₁ + b₁)W₂ + b₂

This is where most of the model's capacity ("knowledge") is stored.

Residual Connections and Layer Normalization

Each sub-layer (attention, FFN) is wrapped with a residual connection (x + sublayer(x)) and layer normalization. Residuals allow gradients to flow directly to earlier layers, enabling training of very deep networks.

Why Transformers Win

• Parallelism - no sequential dependency; entire sequences processed at once on GPUs/TPUs.
• Long-range dependencies - attention directly connects any two tokens regardless of distance.
• Scalability - performance improves predictably as model size and data increase (scaling laws).

Why This Matters in Interviews

The Transformer is the most important architecture in modern AI. Interviewers expect: the self-attention formula and the QKV intuition, why scaling by √dₖ is needed, multi-head attention (different relational subspaces), positional encodings (attention is order-agnostic without them), the encoder/decoder/decoder-only distinction, and residual connections. Explaining why Transformers displaced RNNs (parallelism + long-range attention) is the conceptual anchor.

Common Pitfalls

• Confusing attention scores with probabilities before softmax (they're raw logits).
• Not knowing that decoder-only (GPT-style causal) models are now the dominant variant.
• Forgetting positional encodings - without them, shuffling tokens wouldn't change the output.
• Conflating "the Transformer" with a specific model (it's an architecture; GPT is a model built on it).

References

• Vaswani et al., "Attention Is All You Need" (2017): https://arxiv.org/abs/1706.03762
• Illustrated Transformer (Jay Alammar): https://jalammar.github.io/illustrated-transformer/

More answers