AlgoInsight

Q&A

What is overfitting and underfitting in machine learning?

By AlgoInsight Team2026-06-18machine-learning

Overview

Every ML model sits somewhere on the bias-variance spectrum. Understanding overfitting and underfitting - and how to detect and fix each - is one of the most practically important skills in machine learning.

graph LR
    A[Underfitting] --> B["High bias<br/>Too simple<br/>High train + test error"]
    C[Good fit] --> D["Low bias & variance<br/>Generalizes well"]
    E[Overfitting] --> F["High variance<br/>Too complex<br/>Low train error, high test error"]
    style D fill:#1e1b4b,color:#fff

The Bias-Variance Trade-off

The expected test error decomposes as:

Error = Bias² + Variance + Irreducible noise

Increasing model complexity reduces bias but increases variance. The optimal model minimizes their sum.

Detecting Each Problem

Learning curves

Plot training and validation loss vs training set size (or epochs):

import matplotlib.pyplot as plt
plt.plot(epochs, train_loss, label="Train")
plt.plot(epochs, val_loss,   label="Validation")

The training-vs-validation gap

Scenario Train error Val error Diagnosis
Both high High High Underfitting
Gap widens Low High Overfitting
Both low, close Low Low Good fit

Fixing Overfitting

  1. Get more data - the single most effective remedy; larger datasets leave less room to memorize noise.
  2. Regularization - add a penalty for complexity:
    • L1 (Lasso): loss + λΣ|w| - drives some weights to exactly zero (feature selection).
    • L2 (Ridge/weight decay): loss + λΣw² - shrinks all weights smoothly.
  3. Dropout - randomly zero out neurons during training, forcing redundant representations.
  4. Early stopping - stop training when validation loss starts rising.
  5. Data augmentation - artificially expand training data (flips, crops, noise for images; back-translation for text).
  6. Reduce model complexity - fewer layers/neurons, shallower trees.
  7. Cross-validation - use k-fold CV to get a robust estimate of generalization.
# L2 regularization in PyTorch:
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

# Dropout in a layer:
nn.Sequential(nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.3))

Fixing Underfitting

  1. Increase model capacity - more layers, more neurons, higher-degree polynomial.
  2. Reduce regularization - you may be penalizing the model too much.
  3. Better features - add informative features or feature interactions.
  4. Train longer - more epochs with a suitable learning rate.
  5. More expressive architecture - switch from linear to non-linear model.

The Double-Descent Phenomenon (modern nuance)

With very large deep learning models, the classical bias-variance trade-off is not the whole story. Modern overparameterized models (many more parameters than data points) can interpolate training data (zero training error) and yet generalize surprisingly well - a regime called "benign overfitting." This is why giant language models work so well despite having billions of parameters. The classical advice still applies for moderate-sized models and limited data.

Why This Matters in Interviews

Overfitting/underfitting is asked in nearly every ML interview. Hit: the bias-variance decomposition, the learning-curve diagnostic, and specific fixes for each (regularization/dropout/early stopping for overfitting; more capacity/features for underfitting). Quantitative awareness - "gap between train and val loss signals overfitting" - and knowing when to apply each fix distinguishes a strong answer from a recited definition.

Common Pitfalls

References

More answers