Step 4: Turn Candidates into Preference Pairs

This is the heart of the project: turning candidate responses into preference pairs. This step transforms your raw model outputs into the training signal that DPO uses to align behavior. The quality of your preference labels directly determines the quality of your aligned model—garbage labels produce garbage alignment, no matter how well you tune your hyperparameters.

What makes a good preference judgment?

A good preference judgment is consistent with your alignment rubric and reflects a meaningful quality difference between candidates. When you label response A as better than response B, you're teaching your model to increase the probability of generating responses like A and decrease the probability of responses like B in similar contexts. This means your judgments should be:

• Rubric-aligned: Based on the criteria you defined in Step 1 (helpfulness, safety, honesty, structure, tone)
• Meaningful: The difference between chosen and rejected should be clear enough that the model can learn a generalizable pattern
• Consistent: Similar prompts should receive similar preference judgments, so the model learns stable behavioral patterns rather than noise

If your candidates are nearly identical in quality, it's often better to skip that pair entirely than to force a preference judgment. Training on low-signal pairs wastes compute and can introduce noise that degrades alignment quality.

Three practical routes for labeling preferences

You have three main approaches to creating preference labels, each with distinct trade-offs between quality, speed, and cost:

Route 1: Human labeling

Human labeling produces the highest quality preference data because humans can apply nuanced judgment, understand context deeply, and catch subtle issues that automated systems miss. A human can recognize when a response is technically correct but unhelpful, or when a confident-sounding answer contains a subtle factual error.

The downsides are speed and cost. Even a fast human labeler takes 30–60 seconds per preference pair, meaning 1,000 pairs requires 8–16 hours of focused work. For your first alignment project with 50–200 pairs, this is entirely manageable and highly recommended—you'll develop intuition about what makes responses better that will inform all your future alignment work.

Human labeling also forces you to confront ambiguous cases where neither response is clearly better, which often reveals gaps in your rubric that need clarification.

Route 2: AI-as-a-judge labeling

AI-as-a-judge uses a capable language model (like GPT-4, Claude, or a strong open-source model) to evaluate candidate pairs and select the better response according to your rubric. This approach is fast and scales easily to thousands of pairs—you can label your entire dataset in minutes rather than hours.

The critical requirement is a well-written rubric and prompt that instructs the judge model how to evaluate responses. Your judge prompt should explicitly list your evaluation criteria, provide examples of good vs. bad responses, and ask for structured output (like a JSON object with the chosen response and reasoning). Vague instructions like "pick the better response" produce inconsistent labels.

AI judges work best when the quality differences are clear-cut: factually wrong vs. correct, unsafe vs. safe, helpful vs. unhelpful. They struggle with subtle distinctions in tone, style preferences, or domain-specific quality criteria that require expert knowledge. Always audit a sample of AI-generated labels (at least 50–100 pairs) to verify they match your intended rubric before using them for training.

Route 3: Hybrid approach

The hybrid approach combines the scalability of AI judging with the quality anchoring of human labels. A common pattern: use an AI judge to label 80–90% of your pairs, then have humans label a carefully selected 10–20% subset for quality control.

The human subset should include:

• Cases where the AI judge expressed low confidence or flagged difficulty
• A random sample for general quality auditing
• Edge cases and adversarial prompts where subtle judgment matters most

You can also use human labels to calibrate and improve your AI judge prompt. If you find systematic disagreements between human and AI labels, revise your judge prompt to better capture the human reasoning, then re-label with the improved prompt.

For production alignment at scale, hybrid approaches offer the best balance—you get the throughput of AI judging with confidence that your labels reflect real human preferences.

Practical implementation: Starting with human labeling

For this learning project, we recommend starting with human labeling for your first 50–200 pairs. This builds your intuition and ensures your rubric is well-defined before you attempt to automate it. The workflow below shows a simple, effective pattern you can implement immediately, then extend with AI judging later once you've validated your approach.

Human labeling workflow (simple and effective)

For your first alignment project, manual human labeling is the recommended approach. This workflow is straightforward to implement and provides invaluable learning about what quality actually means in your domain. The process builds your intuition about preference judgments in a way that reading about alignment theory cannot match.

How the workflow operates

The labeling loop is intentionally minimal to reduce friction and keep you focused on making quality judgments rather than wrestling with tooling:

• Load your candidates file (generated in Step 3)
• For each prompt, display both candidate responses (A and B)
• You evaluate both responses against your rubric and select the better one
• The script saves your choice as a preference pair (prompt, chosen, rejected)
• You can skip pairs where neither response is clearly better

This creates a tight feedback loop: you see a prompt, evaluate two responses, make a judgment, and immediately move to the next case. Over 50-100 pairs, patterns emerge. You'll notice recurring failure modes in your base model (rambling, hedging, factual errors) and develop a sharper sense of what "better" means for your specific use case.

Implementation code

import json with open("data/candidates.json", "r", encoding="utf-8") as f:    candidates = json.load(f) prefs = [] print("Labeling instructions:")print("Type A or B to select the better response, or S to skip.\n") for item in candidates:    prompt = item["prompt"]    a = item["A"]    b = item["B"]     print("\nPROMPT:\n", prompt)    print("\nRESPONSE A:\n", a)    print("\nRESPONSE B:\n", b)     choice = input("\nWinner? (A/B/S): ").strip().upper()    if choice == "S":        continue    if choice not in ["A", "B"]:        continue     chosen = a if choice == "A" else b    rejected = b if choice == "A" else a     prefs.append({        "prompt": prompt,        "chosen": chosen,        "rejected": rejected    }) with open("data/preferences.json", "w", encoding="utf-8") as f:    json.dump(prefs, f, indent=2, ensure_ascii=False) print(f"\nSaved {len(prefs)} preference pairs.")

COPY

Why this simple approach is surprisingly powerful

This bare-bones labeling script forces you to define quality through direct comparison rather than abstract criteria. When you see two actual responses side by side, vague rubric items like "be helpful" become concrete decisions: Does this response answer the question directly? Does it hedge unnecessarily? Is the structure clear? Does it guess when it should express uncertainty?

These judgment calls become the training signal for DPO. Every time you choose response A over response B, you're teaching your model to increase the probability of generating responses with A's characteristics and decrease the probability of B's characteristics in similar contexts. The quality of these judgments determines the quality of your aligned model—no amount of hyperparameter tuning can compensate for noisy or inconsistent preference labels.

Human labeling also surfaces edge cases and rubric gaps immediately. You'll encounter pairs where both responses seem equally good (or equally bad), prompts where your rubric doesn't clearly apply, and cases where the "better" response depends on context you didn't consider. These moments are learning opportunities: they tell you where your rubric needs refinement and which types of prompts need more representation in your dataset.

Practical tips for effective labeling sessions

Label in focused sessions of 20-30 pairs at a time, then take a break. Labeling fatigue is real—after an hour of continuous judgments, your consistency degrades and you start making arbitrary choices. Short sessions with breaks maintain judgment quality.

Keep notes on difficult cases. When you encounter a pair where the decision is unclear, write down why. These notes often reveal patterns: maybe you need more guidance on how to handle cases where one response is more complete but the other is more concise, or where one is technically accurate but uses jargon the user might not understand. These observations directly improve your rubric.

Track your skip rate. If you're skipping more than 20-30% of pairs, your candidate generation settings might need adjustment. Too many skips means your temperature gap isn't producing meaningful quality differences, or one temperature setting is consistently producing unusable responses. Adjust your generation parameters and regenerate candidates if needed.

When to graduate to AI-assisted labeling

Once you've manually labeled 100-200 pairs and feel confident in your rubric, you can consider introducing AI-as-a-judge for additional scale. But don't skip the manual phase—those initial human labels become your calibration set for validating that your AI judge actually implements your intended rubric rather than its own implicit preferences.

The discipline of manual labeling also prevents a common failure mode in alignment projects: outsourcing your judgment too early. If you use AI judging before you've internalized what good responses look like, you end up training your model to match an AI judge's preferences rather than your actual alignment goals. Manual labeling grounds your entire pipeline in real human judgment.