Step 8: Improve the Model by Improving the Preferences

This is the most important part of the project: iteration based on systematic evaluation feedback. Alignment is not a one-shot process—it's a cycle of training, evaluating, diagnosing weaknesses, and refining your preference data to address those weaknesses.

When your evaluation results (from both the fixed test set and stress tests) show that your aligned model isn't behaving as desired, resist the temptation to immediately tweak hyperparameters like learning rate or beta. While hyperparameters matter, the quality and coverage of your preference data is almost always the primary bottleneck. Your model can only learn the distinctions you teach it through your chosen vs. rejected pairs.

Diagnosing what to improve:

Look at your evaluation outputs and identify specific failure patterns:

• If the model still hallucinates confidently on ambiguous questions, you need more preference pairs where the "chosen" response admits uncertainty and the "rejected" response fabricates details
• If the model's responses are verbose and unfocused, you need pairs that reward concise, structured answers over rambling ones
• If the model fails to refuse unsafe requests politely, you need pairs demonstrating graceful, firm refusal as "chosen" vs. either compliance or rude rejection as "rejected"
• If the model struggles with a specific type of prompt (e.g., technical explanations, customer service scenarios, or safety edge cases), you're likely underrepresented in that domain—add 10-20 targeted pairs

Strategic improvements to your preference dataset:

• Add more difficult prompts: If your current prompts are too easy, the model learns surface-level patterns. Include prompts that are genuinely challenging—ambiguous contexts, multi-step reasoning, or requests that require careful value judgments. The difficulty should match real-world usage.
• Add pairs where the "rejected" answer is confidently wrong: This is crucial for reducing hallucination. Generate a plausible-sounding but factually incorrect or overconfident response as "rejected," and pair it with a cautious, accurate, or honestly uncertain "chosen" response. This teaches the model that confidence without certainty is worse than admitting limits.
• Add pairs that reward concise clarity over rambling: If your model tends to be verbose, create pairs where "chosen" is a tight, well-organized answer (perhaps 2-3 sentences with clear structure) and "rejected" is the same information buried in unnecessary elaboration. This directly teaches the brevity and focus you want.
• Add pairs that show polite refusal for unsafe requests: Safety alignment requires explicit examples. For each category of harmful request you want to handle (violence, deception, privacy violations, etc.), include at least a few pairs where "chosen" is a respectful, firm refusal with a brief explanation, and "rejected" is either compliance or an awkward/rude rejection.
• Filter noisy labels: Review your existing preference pairs and remove any where the distinction between chosen and rejected is unclear, subjective, or inconsistent with your rubric. Low-quality pairs dilute your signal. If you used AI-as-a-judge, filter out pairs with low confidence scores or where multiple judges disagreed. Quality matters more than quantity.
• Balance your distribution: Check whether your preference pairs are heavily skewed toward certain prompt types or rubric dimensions. If 80% of your pairs test helpfulness but only 5% test safety, your model will be lopsided. Aim for representative coverage across all your rubric criteria.

The iteration cycle:

After identifying gaps from your evaluation, create or collect 20-50 new targeted preference pairs addressing those specific weaknesses. Merge them with your original dataset, retrain with the same hyperparameters (to isolate the effect of data quality), and re-run your fixed evaluation set. Keeping the evaluation prompts constant across iterations is critical—it's the only way to measure whether you're actually improving or just changing behavior randomly.

Document each iteration with:

• What failure pattern you observed (e.g., "model hallucinates technical facts confidently")
• How many pairs you added and what they targeted (e.g., "added 25 pairs where chosen admits uncertainty on technical questions")
• The quantitative or qualitative improvement on your fixed test set (e.g., "4 out of 5 technical prompts now show appropriate hedging vs. 1 out of 5 previously")

This creates a clear improvement trail. If you train three versions and v3 is worse than v2, you can trace back exactly what changed in the data to diagnose why.

When to adjust hyperparameters instead:

Only after you've iterated on data quality and still see issues should you consider hyperparameter changes:

• If the model barely differs from base despite good preference data, try increasing beta (making the preference signal stronger)
• If the model becomes overly repetitive or loses fluency, try decreasing beta or reducing training epochs
• If training is unstable (loss spikes, outputs degrade), try lowering the learning rate

But even then, a single round of thoughtful data improvement usually outperforms extensive hyperparameter tuning. The model learns what you show it—if the preference pairs don't cleanly demonstrate the behavior you want, no amount of tuning will fix that.

Maintaining evaluation consistency:

If you do a second or third training run, keep your evaluation set completely fixed. Never add evaluation prompts to your training data, and never change your evaluation prompts between versions unless you're specifically testing a new capability. Fixed evaluation is the only way to measure true progress rather than just memorization or random variation.

Think of this iteration process as debugging your alignment. Each evaluation reveals bugs (failure modes), and you fix them by patching your dataset (adding targeted examples), not by randomly adjusting settings. This systematic approach is what separates effective alignment from trial-and-error.