MODEL BEHAVIOUR · ALIGNMENT
Large Language Model Alignment and Evaluation: DPO, RLHF and Release Decisions
Measure whether alignment improves the intended behaviour without hiding regressions, safety failures or limitations.
Large language model alignment shapes model behaviour using preference data—comparing chosen and rejected responses—to make the model produce more helpful, safe and appropriate outputs. Direct Preference Optimization (DPO) trains the model directly on preference pairs without a separate reward model. RLHF uses a trained reward model with reinforcement learning. Evaluation must measure task quality, instruction adherence, safety and regressions before a release, revise or reject decision.
Alignment and evaluation concepts
What Does Large Language Model Alignment Change?
Alignment changes the model's output distribution—how it responds to prompts—without changing its underlying knowledge. After fine-tuning, a model may be capable of producing helpful responses but may not consistently do so. Alignment shapes the model's preferences so that, when multiple valid responses exist, it tends to produce the one that humans would choose.
Alignment does not add new knowledge. If the model did not learn a fact during pretraining, alignment cannot make it know that fact. Alignment changes the model's style, tone, refusal behaviour, format consistency and willingness to follow instructions—not its knowledge base. Confusing alignment with knowledge addition is a common misconception that leads to unrealistic expectations.
Alignment can also introduce regressions. If preference data over-weights refusals, the model may become overly cautious and refuse benign requests. If preference data is biased toward a particular style, the model may lose versatility. These regressions must be detected through evaluation, not assumed away.
How Is Preference Data Created?
Preference data consists of triples: a prompt, a chosen response and a rejected response. The two responses are typically generated by the model (or multiple models) with different sampling parameters, then ranked by human annotators or by a stronger model acting as a judge.
Human annotation is the gold standard but is expensive, slow and subject to annotator disagreement. Clear annotation guidelines, annotator training and inter-annotator agreement metrics are essential. The quality of preference data directly determines the quality of alignment—biased or inconsistent annotations produce misaligned models.
Synthetic preference data (generated by a larger model judging outputs of a smaller model) can scale annotation but introduces the risk of inheriting the larger model's biases. A hybrid approach—human annotation for safety-critical dimensions, synthetic for style and format—is common in practice.
How Does Direct Preference Optimization Work?
DPO vs RLHF comparison at the operational level.
| Dimension | DPO | RLHF | |
|---|---|---|---|
| Reward model | Not needed—optimises model directly on preferences | Trained separately on preference data | — |
| Optimisation method | Classification-style loss on preference pairs | PPO (reinforcement learning) with reward model | — |
| Training stability | Generally stable—simpler loss landscape | Can be unstable—PPO is sensitive to hyperparameters | — |
| Compute cost | Lower—single training run | Higher—reward model training + PPO training | — |
| Implementation complexity | Lower—standard supervised training loop | Higher—requires RL infrastructure | — |
| Quality ceiling | Strong for most alignment tasks | Potentially higher for complex preference landscapes | — |
| Risk of reward hacking | Lower—direct preference optimisation | Higher—model can exploit reward model quirks | — |
Where Does RLHF Fit in Model Alignment?
RLHF (Reinforcement Learning from Human Feedback) was the first successful alignment method for large language models, used in the InstructGPT and ChatGPT training pipelines. It has three stages: (1) train a supervised fine-tuned model, (2) train a reward model on preference data, (3) optimise the SFT model using PPO with the reward model as the reward function.
RLHF's strength is its flexibility—the reward model can capture complex preference patterns that a simple classification loss might miss. Its weakness is implementation complexity: PPO is sensitive to hyperparameters, reward models can be exploited (reward hacking), and the training pipeline requires significant infrastructure.
In current practice, DPO has largely replaced RLHF for many alignment tasks due to its simplicity and stability. However, RLHF remains relevant for large-scale alignment where the reward model can be iteratively improved and where the preference landscape is too complex for direct optimisation. The choice between DPO and RLHF is an engineering decision based on the specific alignment goal, available compute and team expertise.
How Should Large Language Model Evaluations Be Designed?
Should the Model Be Released, Revised or Rejected?
Release gate decision framework based on evaluation evidence.
| Decision | Options | Trade-off | Recommendation |
|---|---|---|---|
| Task quality | Improved vs same vs worse than baseline | Improvement justifies release; same may be acceptable; worse requires revision | Release only if task quality improves or stays equal |
| Instruction adherence | Consistently follows instructions vs inconsistent | Inconsistent adherence degrades user experience | Release requires consistent adherence on test set |
| Safety | No new safety failures vs new failures found | New safety failures are a release blocker | Reject if new safety failures; revise if edge cases found |
| Regression count | 0 regressions vs minor vs major | Major regressions in general capabilities are a blocker | Release with 0 major regressions; revise if minor regressions found |
| Known limitations | Documented and acceptable vs undocumented or unacceptable | Undocumented limitations create operational risk | Release only with documented and reviewed limitations |
Alignment failure modes
What goes wrong during alignment and how to detect it.
| Failure | Signal | Cause | Containment | Recovery |
|---|---|---|---|---|
| Over-refusal | Model refuses benign requests that the base model handled | Preference data over-weights refusals; safety annotations too aggressive | Balance preference data; include benign prompts in test set | Reduce alignment strength; re-balance preference data |
| Reward hacking | Model exploits reward model quirks (e.g. verbose answers score higher) | RLHF reward model has exploitable patterns | Monitor output length, format patterns; use DPO to avoid | Retrain reward model; add regularisation; switch to DPO |
| Sycophancy | Model agrees with user even when user is wrong | Preference data rewards agreement over correctness | Include disagree-correctly examples in test set | Re-balance preference data; add correction examples |
| Capability regression | Model scores lower on general benchmarks after alignment | Alignment shifted the model away from its pretrained capabilities | Always evaluate general benchmarks after alignment | Reduce alignment epochs; use KL regularisation; revert and re-align |
Key takeaways
- Alignment shapes behaviour, not knowledge—it changes how the model responds, not what it knows.
- DPO trains directly on preference pairs without a reward model; RLHF uses a reward model with PPO.
- DPO is simpler and more stable; RLHF offers flexibility for complex preference landscapes.
- Evaluation must measure task quality, instruction adherence, safety and regressions.
- Always compare aligned model to baseline on the same test sets.
- Release decisions require evidence across all dimensions, not just task quality.
DPO and RLHF methods are established in their respective papers (Rafailov et al., 2023; Ouyang et al., 2022). Implementation details are verified against Hugging Face TRL documentation.
- Newer alignment methods (KTO, IPO, ORPO) are not covered here but follow related principles.
- Benchmark selection is task-dependent—no universal benchmark suite fits all use cases.
Review cadence: Reviewed every 90 days. Next review by December 2026.
An evaluation score is not yet a release decision. Engineers must connect results, failure modes, safety evidence and limitations to a defensible release, revise or reject recommendation.
In the Large Language Model Course, you move through the same engineering lifecycle with an open-source model: establish a baseline, prepare data, fine-tune with LoRA or QLoRA, evaluate failures, document limitations and make a release, revise or reject decision.
Advanced eight-week live course · Approximately 4–5 hours per week · Python, statistics, machine learning and neural-network foundations expected.
- Tier 1
- Tier 1
- Tier 1
- Tier 1