Q&A

What is the exploding gradient problem?

By AlgoInsight Team2026-06-07

Overview

Neural networks learn via backpropagation, which multiplies gradients layer by layer using the chain rule. If those multiplicative factors are consistently greater than 1, the gradient grows exponentially with depth - it "explodes." The resulting massive updates throw the weights wildly off course, and training diverges. (If the factors are consistently less than 1, you get the opposite: vanishing gradients.)

graph LR
    A[Loss] -->|backprop| B[Layer N gradient]
    B -->|× weight| C[Layer N-1]
    C -->|× weight| D[Layer ...]
    D -->|repeated × > 1| E[Gradient grows exponentially]
    E --> F[Huge weight updates → NaN, divergence]
    style E fill:#1e1b4b,color:#fff

Why It Happens

Backpropagation applies the chain rule, so a gradient flowing back through L layers is a product of L terms (weight matrices and activation derivatives). If the relevant values (e.g. weight magnitudes, the spectral norm of recurrent matrices) are > 1, repeated multiplication blows up:

gradient ∝ w^L      → if w > 1, this explodes as L grows

Recurrent Neural Networks (RNNs) are especially prone because the same weight matrix is applied at every time step, so a long sequence is like a very deep network sharing weights.

Symptoms

• The loss suddenly becomes NaN or Inf.
• The loss oscillates wildly or jumps to huge values instead of decreasing.
• Model weights become very large.
• Training is unstable and never converges.

Solution 1: Gradient Clipping (the standard fix)

Cap the gradient's magnitude before applying the update. Clip by norm rescales the whole gradient vector if its norm exceeds a threshold:

# PyTorch
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

# TensorFlow / Keras
optimizer = tf.keras.optimizers.Adam(clipnorm=1.0)   # or clipvalue=0.5

Clipping preserves the gradient's direction while limiting its size, keeping updates stable.

Other Mitigations

• Proper weight initialization (Xavier/Glorot, He) keeps activations/gradients in a sane range from the start.
• Batch normalization / layer normalization stabilizes the distribution of layer inputs.
• Smaller learning rate reduces the impact of large gradients.
• Better architectures: LSTM/GRU gates regulate gradient flow in RNNs; residual (skip) connections in very deep nets give gradients a direct path, mitigating both exploding and vanishing.
• L2 regularization / weight decay discourages large weights.

Exploding vs Vanishing Gradients

Both stem from the same chain-rule multiplication; the direction of the imbalance determines which one you get.

Why This Matters in Interviews

ML/deep-learning interviews use this to test understanding of backpropagation and training stability. Be able to explain why it happens (chain-rule product of many terms > 1, worsened by weight sharing in RNNs), the symptoms (NaN/divergent loss), and the primary fix (gradient clipping, by norm or value). Contrasting it with vanishing gradients and naming architectural remedies (LSTM, residual connections, normalization, good initialization) signals depth.

Common Pitfalls

• Confusing exploding with vanishing gradients (opposite symptoms and fixes).
• Reaching only for a smaller learning rate when clipping is the targeted fix.
• Ignoring weight initialization, which can cause instability from step one.
• Assuming clipping "fixes" a fundamentally broken model rather than stabilizing training.

References

• Pascanu et al., "On the difficulty of training RNNs" (gradient clipping): https://arxiv.org/abs/1211.5063
• Deep Learning Book (Goodfellow et al.), optimization chapter: https://www.deeplearningbook.org/

More answers