Tuning Large Language Models for Real-World ApplicationsChapter 74

3.4 What Could Go Wrong?

Section 4 of 6-~ 5 min read-Synced from Cuantum content

Common Pitfalls in Reinforcement Learning with Human & AI Feedback

By now, you understand three layers of alignment:

  • Supervised fine-tuning (imitate good answers)
  • Reward modeling (learn what is preferred)
  • Preference optimization (optimize toward better responses)

Reinforcement-based alignment is powerful — but it is also fragile.

Unlike SFT, where mistakes usually show up clearly in outputs, RL-based systems can fail in subtle ways. And subtle failures are more dangerous than obvious ones.

Let’s walk through the most common failure modes.

3.4.1 The Reward Model Learns the Wrong Signal

What happens

The reward model confidently assigns high scores to answers that are verbose, stylish, or confident — but not necessarily correct.

Why it happens

  • Preference data emphasizes tone over correctness.
  • Annotators reward confident language.
  • Rubric does not penalize hallucination strongly enough.

How to fix it

  • Add explicit correctness constraints in the rubric.
  • Include examples where incorrect but confident answers are rejected.
  • Include examples where “I don’t know” is the preferred answer.

Reward models amplify what you measure.

If correctness is not strongly encoded, it will drift.

3.4.2 Reward Hacking

What happens

The policy model learns to exploit weaknesses in the reward model.

For example:

  • Overly long answers score higher.
  • Repetitive phrases increase reward.
  • Safe but unhelpful responses become dominant.

The model learns to “game” the reward function.

Why it happens

The reward model is imperfect.

The policy discovers shortcuts.

How to fix it

  • Add KL regularization to constrain policy updates.
  • Periodically retrain or audit the reward model.
  • Add adversarial evaluation prompts.
  • Monitor qualitative output, not just reward scores.

Reward hacking is not a bug. It is a natural optimization behavior.

3.4.3 Preference Data Is Too Noisy

What happens

Training becomes unstable or inconsistent.

DPO updates push the model in contradictory directions.

Why it happens

  • Annotators disagree.
  • Instructions are ambiguous.
  • Judging rubric is unclear.
  • AI-as-a-judge outputs inconsistent labels.

How to fix it

  • Collect multiple judgments per example.
  • Filter low-confidence judgments.
  • Use structured evaluation prompts.
  • Remove ambiguous prompts from training.

Preference data is powerful — but only if it is coherent.

3.4.4 The Model Becomes Overly Cautious

What happens

After alignment, the model refuses harmless questions or becomes excessively verbose in safety disclaimers.

Why it happens

  • Safety heavily weighted in reward model.
  • Penalization of risk overwhelms helpfulness.
  • Loss aversion in KTO-style objectives.

How to fix it

  • Balance safety and usefulness in the rubric.
  • Add positive examples of safe but direct answers.
  • Penalize unnecessary disclaimers explicitly.

Alignment is a balance.

Too cautious is as problematic as too reckless.

3.4.5 KL Collapse in PPO-Based RLHF

What happens

The model diverges too far from the reference model.

Outputs become unstable or incoherent.

Why it happens

KL penalty is too weak.

PPO tries to maximize reward aggressively.

How to fix it

  • Increase KL coefficient.
  • Reduce learning rate.
  • Monitor KL divergence during training.

If the policy drifts too far from its base distribution, coherence suffers.

3.4.6 DPO Overfits to Preference Data

What happens

Model performs well on training-style prompts but poorly on broader tasks.

Why it happens

  • Preference dataset too narrow.
  • Too many epochs.
  • Beta parameter too large.

How to fix it

  • Lower beta.
  • Reduce epochs.
  • Increase diversity of prompts.

Preference optimization is still supervised learning — it can overfit.

3.4.7 AI-as-a-Judge Drifts Over Time

What happens

Synthetic labels become inconsistent.

Early data and later data follow slightly different standards.

Why it happens

  • Judge prompt modified unintentionally.
  • Judge model version updated.
  • Rubric changes subtly.

How to fix it

  • Freeze judge model version.
  • Keep rubric constant.
  • Maintain a small gold human-labeled validation set.

Judge consistency is critical for long-term alignment loops.

3.4.8 Hidden Distribution Shift

What happens

The aligned model performs well during evaluation but fails in production.

Why it happens

  • Evaluation prompts too similar to training prompts.
  • Real-world user inputs are more diverse.
  • Edge cases were not represented.

How to fix it

  • Create adversarial evaluation sets.
  • Test edge-case prompts.
  • Simulate out-of-distribution queries.

Alignment is contextual.

Real users rarely behave like curated datasets.

3.4.9 The Deep Pattern

Reinforcement-based alignment is more sensitive than supervised fine-tuning.

Why?

Because you are no longer training on fixed answers.

You are training on judgments.

Judgments are:

  • Relative
  • Context-dependent
  • Sometimes subjective
  • Sometimes inconsistent

That is both the strength and the risk.

When debugging RLHF or DPO systems, always ask:

  • What signal is the model actually optimizing?
  • Is that signal aligned with human intent?
  • Is the reward model stable?
  • Is preference data diverse enough?
  • Are we measuring what we care about?

If you can answer those calmly and systematically, you are thinking like an alignment engineer.