Q&A

What are AI evaluation metrics?

By AlgoInsight Team2026-06-18machine-learning

Overview

A model's metric is its target - you optimize what you measure, so measuring the wrong thing produces the wrong model. Each metric captures a different aspect of performance, and most real tasks require considering multiple metrics together alongside business requirements.

graph TD
    A[AI Evaluation Metrics] --> B[Classification]
    A --> C[Regression]
    A --> D[Language Generation]
    A --> E[Language Models]
    B --> B1["Accuracy, Precision, Recall, F1, AUC-ROC"]
    C --> C1["RMSE, MAE, R²"]
    D --> D1["BLEU, ROUGE, BERTScore"]
    E --> E1["Perplexity, MMLU, HumanEval"]
    style A fill:#1e1b4b,color:#fff

Classification Metrics

Confusion Matrix

The foundation of all classification metrics:

                Predicted Positive   Predicted Negative
Actual Positive       TP                   FN
Actual Negative       FP                   TN

Core Metrics

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred))

When to use which:

• Precision matters when false positives are costly: spam filter (mark real email as spam), medical test (unnecessary surgery).
• Recall matters when false negatives are costly: cancer screening (missing a real case), fraud detection (missing fraud).
• F1 when both matter and there's class imbalance.
• Accuracy is misleading with imbalanced classes - a model that predicts all negatives achieves 99% accuracy on a 1% positive dataset.

AUC-ROC

The Area Under the ROC Curve measures ranking ability: how well the model separates classes across all classification thresholds. AUC = 1.0 is perfect; AUC = 0.5 is random. Useful for ranking problems and imbalanced datasets.

Regression Metrics

Language Generation Metrics

BLEU (Bilingual Evaluation Understudy)

Measures n-gram overlap between generated text and reference translations. Standard for machine translation; criticized for not capturing semantic similarity.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

ROUGE-N: n-gram recall between generated summary and reference. ROUGE-L: longest common subsequence. Standard for summarization.

BERTScore

Uses contextual BERT embeddings to measure semantic similarity between generated and reference text - captures meaning beyond surface n-gram overlap.

Perplexity

For language models: how "surprised" the model is by held-out text. Lower = better.

Perplexity = exp(-1/N × Σ log P(xₜ | x₁...xₜ₋₁))

LLM Benchmarks

LLM-as-Judge

Use a powerful LLM (GPT-4, Claude) to evaluate generated outputs on criteria like helpfulness, accuracy, safety, and reasoning quality. More aligned with human judgement than BLEU/ROUGE for open-ended generation.

Why This Matters in Interviews

Metric questions are fundamental in every ML interview. The core signals: know the confusion matrix; explain precision vs recall and when each matters (medical/fraud = recall; spam filter = precision); why accuracy is misleading with class imbalance; AUC-ROC for ranking; RMSE vs MAE; and modern LLM evaluation (MMLU, LLM-as-judge). Raising "choose the metric that matches your business cost" shows engineering maturity.

Common Pitfalls

• Using accuracy with imbalanced classes - a naive classifier can achieve 99% accuracy on a 1% fraud dataset.
• Optimizing a metric that doesn't match the business problem.
• Reporting only training metrics, not validation/test metrics.
• Using BLEU for open-ended generation - it's a poor proxy for quality.
• Threshold confusion: precision/recall depend on the classification threshold; AUC is threshold-independent.

References

• scikit-learn metrics docs: https://scikit-learn.org/stable/modules/model_evaluation.html
• MMLU benchmark: https://arxiv.org/abs/2009.03300

More answers