Nestor G Pestelos Jr · Reference · Print this page

Artificial Intelligence · Model Alignment

Reinforcement Learning from Human Feedback (RLHF)

Reference entry · last updated September 11, 2026 · Previous version (20260911)

Reinforcement learning from human feedback (RLHF) uses human evaluations to guide reinforcement learning. In a common language-model pipeline, a reward model learns from human comparisons, and the language model is optimized against that reward.[1, 4]

1. First Principles: Preference as the Training Signal

RLHF uses human feedback to learn objectives that are difficult to specify directly. In preference-based RLHF, people compare outputs or behavior, and a reward model learns from those judgments.[1]

A common preference model assumes a latent reward function \(r^*(x, y)\) over prompt \(x\) and response \(y\), and models a human's pairwise choice with the Bradley-Terry model: given a preferred response \(y_w\) and a rejected response \(y_l\), the probability that a rater picks \(y_w\) is[2, 8]

where \(\sigma\) is the logistic function. Only reward differences are identifiable, so an additive constant is free, and the model treats disagreement between raters as logistic noise around a single shared utility.

Given a reward function, a common language-model objective adds a Kullback-Leibler (KL) penalty to expected reward. The tuned policy \(\pi_\theta\) is penalized for moving away from a reference policy \(\pi_{\text{ref}}\), usually the supervised fine-tuned model:[4, 8]

This is a KL-regularized objective: divergence has a cost rather than a hard upper bound. For a fixed reward scale, a larger \(\beta\) penalizes departure from the reference policy more strongly. A smaller \(\beta\) permits more departure, which can increase exposure to reward-model errors; neither setting guarantees fluency or diversity.[8, 9]

2. The Standard Three-Stage Pipeline

The InstructGPT pipeline runs in three stages: supervised fine-tuning, reward model training, and reinforcement learning against that reward.[4]

2.1 Supervised Fine-Tuning

A base model is first fine-tuned on curated prompt-response demonstrations so that it follows the conversational schema and produces on-task answers. This model becomes both the starting point for reinforcement learning and the reference policy \(\pi_{\text{ref}}\) in the KL term.[4]

2.2 Reward Modeling

Human labelers rank or compare multiple sampled responses to each prompt. A reward model \(r_\psi(x, y)\), typically the SFT model with a scalar output head, is trained on these comparisons by minimizing the negative log-likelihood of the Bradley-Terry model:[3, 4]

The reward model compresses a finite set of human comparisons into a function that can score any response. Its accuracy on held-out comparisons, and its behavior far from the data it was trained on, affect the quality of the optimized policy.

2.3 Policy Optimization with PPO

The policy is optimized against the reward model with Proximal Policy Optimization (PPO) in the InstructGPT pipeline.[4, 6] A per-token KL penalty is folded into the reward, giving this regularized reward for a full response:

Token generation is treated as a sequential decision process: the state is the prompt plus the tokens produced so far, each action is the next token, and the environment transition is deterministic concatenation. PPO's clipped surrogate objective discourages large policy updates; see reinforcement learning for that objective in full. A separate value network estimates the advantage, which adds memory and tuning cost.

3. Direct Preference Optimization

Direct Preference Optimization (DPO) removes the separate reward model and the online sampling loop.[8] Rafailov et al. showed that the optimal policy for the KL-regularized objective corresponds to an implicit reward

where \(Z(x)\) is a partition function that cancels inside a Bradley-Terry difference. Substituting this parameterization into the preference loss gives an objective in the policy alone, trained on a fixed dataset of comparison pairs with a binary cross-entropy loss:

Rafailov et al. evaluated DPO on sentiment control, summarization, and single-turn dialogue. DPO outperformed PPO on sentiment control and matched or improved response quality on the other two tasks in their experiments.[8]

Standard DPO trains on a fixed preference dataset. PPO-based RLHF samples new responses during optimization and scores them with a reward model that can remain fixed. Collecting new human comparisons is a separate labeling step.[4, 8]

4. Reward Model Overoptimization

A reward model approximates human judgment. Strong optimization can exploit its errors: the proxy score may rise while the quality it is intended to measure falls.[9]

Gao et al. studied this using a larger "gold" reward model to generate synthetic preferences and evaluate policies trained against smaller proxy models. Gold-model reward often rose and then fell as optimization increased. Their fitted relationships depend on the optimization method, reward-model size, and preference-data size. The gold model was a synthetic stand-in for human judgment.[9]

A KL penalty and early stopping can limit optimization against a flawed reward model. These controls reduce exposure to errors without establishing that the reward captures human preferences.[9, 12]

5. Reinforcement Learning from AI Feedback

Collecting human comparisons can be expensive and slow. Reinforcement learning from AI feedback (RLAIF) replaces some or all of the human labels with preferences generated by another language model.[10, 11]

Constitutional AI applies this to harmlessness: the model critiques and revises its own responses against a written list of principles, then a preference model is trained on AI-labeled comparisons and used for the RL stage, keeping human input mostly at the level of the principles.[10] Lee et al. compared RLAIF against RLHF on summarization and dialogue and reported comparable human win rates, along with a variant that scores responses directly with an off-the-shelf model instead of training a separate reward model.[11] AI feedback inherits the biases and blind spots of the labeling model.

6. Limitations and Open Problems

Casper et al. survey the failure modes of RLHF and separate problems that better engineering can address from ones that are structural to the approach.[12] The recurring issues:

• Feedback quality. Raters disagree, make errors, apply inconsistent standards, and can be misled by fluent or confident-sounding answers. Labeler pools are small and not representative of the eventual user base.
• Reward misspecification. Pooling diverse raters into one reward model can hide conflicting preferences. The Bradley-Terry model imposes assumptions that real preferences can violate.[12]
• Sycophancy. Human preference data can reward agreement with a user's beliefs. Optimizing against such preferences can increase sycophancy, sometimes at the expense of truthfulness.[13] See sycophancy.
• Distribution shift and reward hacking. The policy is optimized on its own shifting output distribution, which pulls it toward regions where the reward model is unreliable and exploitable. See reward hacking.
• Diversity and calibration. Preference optimization can reduce output variety.[12] OpenAI reported poorer calibration after GPT-4 post-training on a subset of MMLU.[14] See fine-tuning for alignment trade-offs.
• Systemic cost. The full pipeline needs demonstration data, a labeling operation, a trained reward model, and a stable distributed RL setup, which concentrates the method among well-resourced labs.

7. Development

• 2017. Christiano et al. train deep RL agents on simulated robotics and Atari from human comparisons of short trajectory clips, with humans labeling under one percent of interactions.[1]
• 2019. Ziegler et al. apply preference-based reward learning with a KL penalty to language models, fine-tuning GPT-2 for continuation style and summarization.[5]
• 2020. Stiennon et al. use RLHF for summarization, with tuned summaries preferred to human reference summaries on Reddit TL;DR.[7]
• 2022. Ouyang et al. put RLHF at the center of instruction following; outputs from a 1.3B InstructGPT model are preferred to those of 175B GPT-3.[4] Bai et al. apply it to a combined helpful and harmless assistant with iterated online training and red-teaming.[3]
• 2022 to 2024. DPO removes the online RL loop,[8] Gao et al. quantify overoptimization,[9] and Constitutional AI and RLAIF shift much of the labeling to models.[10, 11]