Q&A

What is Generative AI and how do diffusion models work?

By AlgoInsight Team2026-06-18system-design

Overview

Generative models have always existed (GANs, VAEs), but the 2020s saw a step change: diffusion models for images/video/audio, and autoregressive Transformers for text. Together they define "generative AI." Modern multimodal models (GPT-4o, Gemini) combine both - generating text, images, and audio from a unified model.

graph LR
    A["Pure noise<br/>(Gaussian)"] --> B["Reverse diffusion<br/>T denoising steps"]
    B --> C["Generated image<br/>conditioned on text"]
    D["Real image"] -->|"Forward: add noise"| A
    style B fill:#1e1b4b,color:#fff

How Diffusion Models Work

Forward Process (Training signal)

Gradually add Gaussian noise to a real image over T steps (e.g. T=1000):

x₁ = x₀ + ε₁, x₂ = x₁ + ε₂, ..., xₜ ≈ N(0, I)

At step T the image is pure noise. A closed-form formula gives xₜ directly from x₀:

xₜ = √(ᾱₜ) x₀ + √(1-ᾱₜ) ε,   ε ~ N(0,I)

Reverse Process (Generation)

A neural network (typically a U-Net) is trained to predict the noise ε that was added at each step, given the noisy image xₜ and the timestep t:

ε_θ(xₜ, t) ≈ ε

At generation time, start from pure noise and run the reverse process - the network iteratively denoises, recovering a coherent image.

Text Conditioning (for text-to-image)

A text encoder (often CLIP or a T5 encoder) converts the prompt into embeddings. These condition the U-Net's denoising via cross-attention layers: at each denoising step, the network attends to the text embedding to guide "what" to generate.

Classifier-Free Guidance (CFG)

Control the trade-off between prompt fidelity and image diversity:

ε_guided = ε_uncond + cfg_scale × (ε_cond - ε_uncond)

Higher CFG scale → images more tightly match the prompt (less creative variety); lower → more diverse.

Key Generative AI Models

Latent Diffusion (Stable Diffusion)

Running diffusion in pixel space is computationally expensive. Latent diffusion (Rombach et al., 2022) runs the diffusion process in a compressed latent space using a variational autoencoder (VAE):

1. VAE encoder compresses the image to a smaller latent.
2. Diffusion happens in latent space (8–16× smaller).
3. VAE decoder reconstructs the full image from the denoised latent.

This makes high-resolution image generation tractable on consumer hardware.

GANs vs Diffusion Models vs VAEs

Diffusion models win on quality and stability; GANs still win on inference speed for some applications.

Why This Matters in Interviews

Generative AI questions are very common in 2025–26. Cover: the forward (add noise) vs reverse (denoise) process, U-Net as the denoising network, text conditioning via cross-attention, classifier-free guidance, and why latent diffusion is efficient (compressed space). Comparing diffusion to GANs (training stability, quality/diversity advantages) and naming current models shows currency.

Common Pitfalls

• Confusing the forward process (data → noise, training signal) with the reverse process (noise → image, generation).
• Thinking diffusion models "store" images - they learn a general denoising function.
• Conflating "generative AI" with only LLMs - it covers all generative models (images, audio, video, code).
• Not knowing about latent diffusion and why pixel-space diffusion is impractical at high resolution.

References

• Ho et al., "Denoising Diffusion Probabilistic Models" (2020): https://arxiv.org/abs/2006.11239
• Rombach et al., "Latent Diffusion Models" (2022): https://arxiv.org/abs/2112.10752

More answers