Step 3: Generate Candidate Responses

For each prompt, you need at least two candidate responses so you can create a preference pair. This is the fundamental requirement for DPO training: the algorithm learns by comparing responses and understanding which one better aligns with your goals. Without multiple candidates, there's no comparison to make and no preference signal to learn from.

Why multiple candidates matter

The quality of your DPO alignment depends heavily on the diversity and contrast between candidate responses. If your two candidates are nearly identical, the preference signal is weak—the model learns very little about what makes a response better. Ideally, your candidates should exhibit meaningful differences in tone, structure, accuracy, or safety that allow you to demonstrate your alignment priorities clearly.

Temperature is your primary tool for creating this diversity. Higher temperature values (like 0.9 or 1.0) produce more creative and varied responses, while lower values (like 0.3 or 0.6) produce more conservative and predictable outputs. By generating one response at low temperature and another at high temperature, you naturally create contrasting candidates that often differ in exactly the dimensions you care about—safety, confidence calibration, verbosity, and creativity.

You can generate candidates in two common ways:

Option A: Candidate responses from your base model

This is the most realistic alignment workflow and the recommended approach for this project. You generate multiple responses from the same base model you intend to align, using different sampling parameters (typically different temperature values) to create diversity.

Why this works well: The candidates represent the natural range of behaviors your base model is capable of. When you label preferences between these candidates, you're essentially teaching the model to favor certain parts of its existing behavioral distribution over others. This is exactly what DPO is designed to do—shift probability mass toward preferred behaviors and away from dispreferred ones, without introducing behaviors the model can't already produce.

This approach also creates realistic training data. The "rejected" responses aren't artificially bad—they're responses your model would actually generate in production if you deployed it unaligned. Learning to avoid these realistic failure modes is much more valuable than learning to avoid synthetic or exaggerated bad examples.

Option B: Candidate responses from a stronger teacher model

In this approach, you generate candidate responses from a more capable model (like GPT-4, Claude, or a larger open-source model), then use these as training targets for your smaller base model. One response might come from your base model, while the "better" response comes from the teacher.

When this is useful: If your base model is extremely weak or poorly instruction-tuned, it may not be capable of generating good responses even with optimal sampling parameters. In these cases, using a stronger teacher provides a quality ceiling your base model can learn to approach through DPO.

The trade-off: This can reduce realism and sometimes leads to distribution mismatch problems. If your teacher model is much more capable than your base model, the "chosen" responses may contain reasoning patterns, knowledge, or linguistic capabilities your base model fundamentally cannot learn to reproduce. The model may learn to imitate the surface features of good responses without understanding the underlying quality. You're also not learning from your base model's actual failure modes—you're learning from an artificial comparison that may not reflect how users will experience your deployed model.

For this learning project, stick with Option A. It's simpler, requires only one model, and teaches you the core DPO mechanics without the confounding variable of cross-model distillation.

Implementation: Generating candidates from your base model

Below is Option A implemented in code. This script loads your base model, generates two candidate responses per prompt using different temperature values, and saves the results for labeling.

import jsonimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer MODEL_NAME = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto") def generate(prompt, temperature=0.7, top_p=0.9, max_new_tokens=200):    formatted = f"### Instruction:\n{prompt}\n### Response:\n"    inputs = tokenizer(formatted, return_tensors="pt").to(model.device)    with torch.no_grad():        out = model.generate(            **inputs,            do_sample=True,            temperature=temperature,            top_p=top_p,            max_new_tokens=max_new_tokens        )    return tokenizer.decode(out[0], skip_special_tokens=True) with open("data/prompts.json", "r", encoding="utf-8") as f:    prompts = json.load(f) candidates = []for p in prompts:    a = generate(p, temperature=0.6)    b = generate(p, temperature=0.9)    candidates.append({"prompt": p, "A": a, "B": b}) with open("data/candidates.json", "w", encoding="utf-8") as f:    json.dump(candidates, f, indent=2, ensure_ascii=False)

COPY

Understanding the temperature choice

This code generates response A with temperature=0.6 (more focused and conservative) and response B with temperature=0.9 (more creative and varied). This temperature gap of 0.3 typically produces meaningfully different responses without making either response completely random.

The lower-temperature response tends to be safer, more predictable, and sometimes more accurate because it sticks to high-probability tokens. The higher-temperature response is more diverse and creative but also more prone to hallucination, rambling, or tone inconsistencies. When you label these pairs, you're often choosing between conservative-but-boring versus creative-but-risky, which teaches your model exactly how to balance these trade-offs according to your rubric.

You can experiment with different temperature values if the default gap doesn't produce enough contrast. Some practitioners use temperatures as low as 0.3 and as high as 1.2 to maximize diversity. Just be careful not to make the high-temperature responses so chaotic that they're never preferred—you want both candidates to be plausible choices, with clear reasons why one is better.

Output format and next steps

You now have two responses per prompt, saved in a structured JSON format. Each entry contains the original prompt and both candidate responses (labeled A and B). This file becomes the input for your preference labeling workflow in Step 4, where you'll decide which response better matches your alignment goals and convert these candidates into the preference pairs DPO needs for training.

Before moving to labeling, it's worth spot-checking your candidates file. Load it and read through 10-15 examples. Do the candidates actually differ in meaningful ways? Are both candidates coherent and plausible, or is one temperature setting producing consistently poor responses? If the candidates are too similar, increase your temperature gap. If the high-temperature candidates are consistently nonsensical, reduce the upper temperature. The goal is to create labeling decisions that feel meaningful—where you're genuinely choosing the better response based on your rubric, not just rejecting obvious garbage.