AlgoInsight

Q&A

What is Reinforcement Learning from Human Feedback (RLHF)?

By AlgoInsight Team2026-06-18

Overview

A pretrained LLM knows language but has no understanding of what constitutes a good response - it might complete a story when asked to answer a question, or give harmful instructions because they appear in training data. RLHF teaches the model human preferences: what makes a response helpful, honest, and harmless, using human judgements as the training signal.

graph TD
    A["Phase 1: SFT<br/>Fine-tune on human-written demos"] --> B["Phase 2: Reward Model<br/>Train on human pairwise comparisons"]
    B --> C["Phase 3: PPO<br/>Fine-tune LLM to maximize reward"]
    C --> D["Aligned assistant<br/>Helpful, harmless, honest"]
    style C fill:#1e1b4b,color:#fff

Phase 1: Supervised Fine-Tuning (SFT)

Human labellers write high-quality responses to a diverse set of prompts. The base LLM is fine-tuned on these (prompt, response) pairs with standard cross-entropy loss:

L_SFT = -log P(response | prompt)

This teaches the model the format and style of helpful responses. The SFT model is still not optimal - it simply imitates the demonstrations.

Phase 2: Reward Model Training

Human labellers are shown pairs of responses to the same prompt and choose which is better. These comparisons train a reward model (RM) - typically the SFT model with a linear head replacing the language model head:

RM(prompt, response) → scalar reward score

The reward model is trained to assign higher scores to preferred responses:

L_RM = -log(σ(RM(prompt, r_better) - RM(prompt, r_worse)))

Now we have a function that estimates how much a human would prefer any given response.

Phase 3: PPO Fine-Tuning

The SFT model is further fine-tuned using Proximal Policy Optimization (PPO), a reinforcement learning algorithm, to maximize the reward model's score while staying close to the SFT model (via a KL penalty):

L_PPO = reward(prompt, response) - β × KL(π_RL || π_SFT)

The KL penalty prevents reward hacking - without it, the model might find ways to produce high-reward outputs that sound unnatural or diverge far from the SFT model (e.g. repeating "I love you" to game a sycophantic reward model).

RLAIF - RL from AI Feedback (Constitutional AI)

Labelling at scale is expensive. RLAIF (Anthropic) replaces human comparisons with AI-generated comparisons: a powerful LLM evaluates output pairs against a written constitution of principles and generates synthetic preference data to train the reward model. This scales alignment without proportional human labelling cost.

DPO - Direct Preference Optimization

A recent simplification (Rafailov et al., 2023): directly optimize the LLM on preference pairs without a separate reward model or PPO. Mathematically equivalent to RLHF under certain conditions, but much simpler to implement and more stable:

# DPO fine-tuning: preferred response vs rejected response
# No separate RM or RL loop needed

DPO has largely replaced PPO-based RLHF for many practitioners due to simplicity.

What RLHF Achieves (and Doesn't)

Achieves:

Doesn't fully solve:

Why This Matters in Interviews

RLHF is the standard LLM alignment technique and appears in most AI/ML interviews at product or research companies. Cover: the three phases (SFT → RM → PPO), the KL penalty rationale (prevent reward hacking), RLAIF as the scalable variant, and DPO as the simpler modern alternative. Understanding that RLHF teaches preferences - not truth - and that reward hacking and sycophancy are inherent risks shows real depth.

Common Pitfalls

References

More answers