Step 2: Create Prompts for Preference Data

Section 3 of 19-~ 3 min read-Synced from Cuantum content

DPO requires prompts that represent what users will actually ask. This is crucial because your model will learn to align its behavior specifically on the distribution of prompts you provide during training. If your training prompts are artificial or don't match real usage patterns, your aligned model may perform well on toy examples but fail in production when users ask questions in different styles or domains.

Think of your prompt set as defining the operational envelope of your alignment. The model will learn preference patterns within this distribution, but those patterns may not generalize well to prompt types it hasn't seen during DPO training. This is why diversity in your prompt set matters just as much as quantity.

Build a prompt set

Start with 200–1,000 prompts for a production-quality alignment run. For learning the pipeline and validating your workflow, even 50 carefully chosen prompts is fine—you'll iterate faster and can scale up once you've proven the process works.

Your prompts should span the variety of use cases your chatbot will encounter. If you're building a customer support bot, include polite inquiries, frustrated complaints, unclear questions, and edge cases. If you're building a coding assistant, include conceptual questions, debugging scenarios, code explanation requests, and implementation tasks. The goal is coverage across your expected usage patterns, not just easy or common cases.

A practical strategy: start by collecting real user queries if you have them, or simulate realistic queries by putting yourself in your users' shoes. Avoid the temptation to make prompts too clean or well-structured—real users write ambiguous, typo-ridden, or poorly formatted prompts, and your model needs preference data that teaches it how to handle these gracefully.

Examples:

"Explain gradient accumulation in simple terms."
"Write a polite refund reply for this customer message…"
"Summarize this paragraph in one sentence…"
"Give troubleshooting steps for a login issue…"

Notice the variety here: conceptual explanation, customer service tone, summarization, and procedural help. Each prompt type will generate different candidate responses and teach your model different aspects of your alignment rubric. The conceptual prompt tests clarity and honesty about uncertainty. The customer service prompt tests tone and empathy. The summarization prompt tests instruction-following and conciseness. The troubleshooting prompt tests structure and helpfulness.

Save them as a JSON file:

data/prompts.json

[  "Explain gradient accumulation in simple terms.",  "Write a calm customer support reply: 'My package arrived damaged.'",  "Summarize: Instruction tuning improves prompt following."]

This simple JSON array structure makes it easy to load, iterate over, and expand as you refine your dataset. You can add prompts incrementally, organize them by category in separate files, or version them as your alignment goals evolve. The format is deliberately minimal—just strings—because the complexity lives in choosing prompts that truly represent your use case, not in the data structure itself.