Tuning Large Language Models for Real-World ApplicationsChapter 11

1.1 Creating & Curating Instruction Datasets

Section 1 of 6-~ 54 min read-Synced from Cuantum content

Large Language Models (LLMs) begin their lives as powerful but somewhat unfocused learners. During pretraining, they absorb vast quantities of text from books, websites, research papers, code repositories, and many other sources. This process gives them a broad understanding of language, knowledge, and patterns. However, pretraining alone does not make a model particularly good at following human instructions.

If you ask a purely pretrained language model to summarize an article, write Python code, or answer a question step by step, the results may feel inconsistent. Sometimes the model behaves exactly as expected; other times it generates irrelevant text, incomplete answers, or responses that do not follow the structure requested by the user.

This is where instruction tuning enters the picture.

Instruction tuning transforms a general language model into a helpful assistant that understands how to respond to human prompts. Instead of simply predicting the next word in a sequence of internet text, the model learns to interpret instructions and produce useful responses. In other words, it learns how to behave in an interactive setting.

One of the most important techniques used to accomplish this transformation is Supervised Fine-Tuning (SFT).

In SFT, we take a pretrained model and train it further on a curated dataset of instruction–response pairs. Each example shows the model what a good response looks like for a particular type of instruction. Over time, the model learns the patterns behind these examples and generalizes them to new tasks.

A typical SFT training sample might look like this conceptually:

Instruction:

Explain the difference between a list and a tuple in Python.

Response:

A list in Python is mutable, meaning its elements can be modified after creation. A tuple, on the other hand, is immutable, meaning its contents cannot be changed once defined...

By training on thousands—or sometimes millions—of these examples, the model gradually learns how to produce helpful, structured responses.

In modern LLM development pipelines, instruction tuning usually represents the first alignment step after pretraining. It is often followed by more advanced alignment techniques such as Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), which further refine the model's behavior.

However, none of those later stages would be possible without a solid foundation of high-quality instruction data.

That is why the most important component of any SFT pipeline is not the training code, the GPUs, or even the model architecture.

It is the dataset.

The quality, diversity, and structure of the instruction dataset largely determine how capable the resulting model will be. A poorly curated dataset leads to a model that gives shallow or confusing answers. A carefully designed dataset can produce a model that feels remarkably helpful and intelligent.

In this chapter, we will explore the process of building such datasets.

We will examine how instruction data is created, how it is curated, how examples are formatted, and how modern LLM pipelines transform this data into training material.

We begin with the most fundamental question: Where do instruction datasets come from?

Instruction datasets are the foundation of supervised fine-tuning. Without them, an LLM cannot learn how to translate human intent into structured responses. They serve as the bridge between a model's raw language understanding capabilities and its ability to serve as a useful, responsive assistant.

Creating these datasets is both a technical process and a design challenge. It requires thinking carefully about the types of tasks the model should perform, the quality of the responses it should produce, and the diversity of instructions it should understand. Dataset creators must balance several competing concerns: breadth versus depth, quality versus quantity, and coverage of common tasks versus handling of edge cases.

The design process begins with fundamental questions: What should the model be capable of doing? How should it respond to ambiguous requests? What tone should it adopt? Should it refuse certain types of requests? Each of these questions influences how the dataset is constructed.

Modern instruction datasets are rarely created through a single method. Instead, they typically emerge from a combination of sources:

  • Human-written examples
  • AI-generated synthetic data
  • Existing task datasets
  • Community contributions
  • Conversation logs
  • Expert annotations

Each source has its strengths and weaknesses. A well-designed instruction dataset often combines several of them to maximize both coverage and quality.

Human-written examples provide the highest quality and most natural phrasing, but are expensive and slow to produce at scale. AI-generated synthetic data offers rapid scaling but risks introducing artifacts or biases from the generating model. Existing task datasets provide well-tested examples but may lack the conversational tone users expect from an assistant. Community contributions can offer diverse perspectives but require careful moderation. Conversation logs reflect real user needs but may contain noise, errors, or inappropriate content. Expert annotations ensure technical accuracy but may be too formal or specialized for general use.

The art of dataset construction lies in knowing which sources to combine, how to balance them, and how to filter and refine the results. A dataset built entirely from synthetic data may produce a model that feels artificial or repetitive. A dataset built only from human examples may lack the scale needed for robust performance. The most effective approaches blend multiple sources strategically, using each where it provides the greatest value.

Before exploring these sources in detail, it is helpful to understand the basic structure of instruction training examples. The format of these examples shapes how the model learns to interpret and respond to instructions, making it one of the most important design decisions in the entire pipeline.

1.1.1 Structure of an Instruction Example

At its core, instruction tuning relies on paired data—examples that explicitly connect what a user asks for with what the model should produce in response.

Each training sample typically contains three elements:

  • Instruction
  • Input (optional)
  • Response

Understanding how these elements work together is essential for anyone building or working with instruction datasets. Let's examine each component in detail.

Instruction

The instruction describes the task the model should perform. It is written in natural language and resembles a prompt a user might type into an AI assistant.

Instructions can range from simple commands to complex multi-step requests. They may be direct ("Translate this sentence") or more open-ended ("Help me understand this concept"). The key characteristic is that they communicate intent—they tell the model what kind of output is expected.

Examples:

  • Summarize the following paragraph.
  • Translate this sentence into German.
  • Write a Python function that calculates factorial.
  • Explain gradient descent in simple terms.

Notice the variety in these examples. Some ask for transformations of existing content, others request original creations, and still others seek explanations. A well-rounded instruction dataset captures this diversity, teaching the model to recognize and respond appropriately to different types of requests.

The phrasing of instructions also matters. Real users don't always phrase requests consistently. Some might say "Summarize this," while others say "Give me a summary of the following text" or "What are the main points here?" A robust dataset includes variations in phrasing so the model learns to understand intent regardless of exact wording.

Input (Optional)

Some tasks require additional context beyond the instruction itself. This context is provided in the input field.

For example, a summarization task must include the text to summarize. A translation task must include the source sentence. A code debugging task might include the broken code. A question answering task might include a passage containing the answer.

Not all tasks require separate input. Instructions like "Write a haiku about autumn" or "Explain photosynthesis" are self-contained—they don't need additional context to be answerable. In these cases, the input field is left empty.

The distinction between instruction and input helps maintain clarity in the dataset structure. The instruction tells the model what to do, while the input provides the material to work with.

Example:

Instruction:

Summarize the following paragraph.

Input:

Large language models are trained on massive text datasets and learn statistical patterns in language. These models can generate coherent responses to prompts and perform a wide range of tasks including translation, summarization, and question answering.

Response:

Large language models learn language patterns from large datasets and can perform tasks such as translation, summarization, and question answering.

In this example, the instruction establishes the task type (summarization), the input provides the source material, and the response demonstrates the expected output.

Response

The response represents the ideal output the model should generate given the instruction and input.

During SFT training, the model learns by predicting this response token by token. The training objective is straightforward: maximize the likelihood of generating the exact tokens in the response field, conditioned on the instruction and input that precede it.

This is where the quality of the dataset becomes critical. If responses are poorly written, factually incorrect, or unhelpful, the model will learn to produce similar low-quality outputs. Conversely, well-crafted responses teach the model not just what to say, but how to say it—with appropriate structure, tone, and level of detail.

Response quality encompasses several dimensions. Each dimension plays a critical role in determining whether the model learns to behave as a helpful, reliable assistant. Understanding these dimensions helps dataset creators make informed decisions about which responses to include, which to revise, and which to discard entirely.

  • Correctness: The response should accurately fulfill the instruction. This is the most fundamental quality requirement. If a user asks for the capital of France, the answer must be Paris, not London. If they request a Python function to sort a list, the code must actually perform that operation without errors. Correctness is non-negotiable—teaching a model incorrect information undermines its reliability and trustworthiness. For factual questions, this often requires verification against authoritative sources. For tasks like code generation, it may involve running the code to confirm it executes properly. For reasoning tasks, it means ensuring the logic is sound and the conclusion follows from the premises.
  • Completeness: It should provide sufficient information without being unnecessarily verbose. A response that's too brief may leave the user confused or force them to ask follow-up questions. A response that's too long may overwhelm them with irrelevant details or waste their time. The ideal response contains exactly what's needed to fulfill the instruction—no more, no less. This balance varies by context. A technical explanation might require more depth than a simple factual answer. A beginner's question might need more background than an expert's query. Good responses anticipate what the user needs to know and provide appropriate detail for the context.
  • Clarity: The language should be clear and easy to understand. Even a correct and complete response fails if the user cannot comprehend it. Clarity involves using straightforward language, organizing information logically, and avoiding unnecessary jargon or complexity. When technical terms are necessary, they should be explained or defined. Sentences should be well-structured, and ideas should flow naturally from one to the next. The goal is to minimize cognitive effort for the reader while maximizing their understanding. A clear response demonstrates respect for the user's time and attention.
  • Tone: The response should match the expected style (professional, conversational, technical, etc.). Different contexts demand different tones. A customer service interaction might require warmth and empathy. A technical documentation request might call for precision and formality. A creative writing prompt might benefit from playful or expressive language. The tone should align with both the instruction itself and the broader context in which the model will be deployed. Consistency in tone across similar types of requests helps users develop accurate expectations about how the model will respond. Inappropriate tone—being too casual in a formal context or too stiff in a friendly one—can undermine the user's confidence in the response even when the content is correct.
  • Safety: It should avoid harmful, biased, or inappropriate content. This dimension has become increasingly important as language models are deployed in real-world applications. Responses must not promote violence, hatred, illegal activities, or dangerous behaviors. They should not contain offensive language or perpetuate harmful stereotypes. They should respect privacy and avoid generating content that could be used to manipulate or deceive. When instructions request inappropriate content, the response should politely decline rather than comply. Safety considerations also extend to subtler issues like fairness and representation—ensuring that responses don't systematically favor or disadvantage particular groups. Building safety into the training data is far more effective than trying to patch problems afterward through content filtering alone.

These quality dimensions often require human judgment or careful review to ensure. Automated metrics can catch some issues, but subtle problems with tone, helpfulness, or appropriateness typically require human evaluation.

Data Representation

A typical JSON representation of instruction examples looks like this:

{  "instruction": "Explain the concept of overfitting in machine learning.",  "input": "",  "output": "Overfitting occurs when a model learns the training data too closely, including noise and random fluctuations, which reduces its ability to generalize to new data."}

Many popular instruction datasets, such as Alpaca, Dolly, and FLAN, follow similar structures. This standardization makes it easier to combine datasets from different sources and to build training pipelines that work across multiple data sources.

The JSON format is convenient for storage and processing, but it's not the format the model actually sees during training. Instead, these structured fields must be converted into a continuous text sequence that the language model can process.

Prompt Formatting

During training, the separate instruction, input, and response fields are combined into a single formatted prompt. The exact formatting can vary, but most approaches use special markers or structured templates to delineate different sections.

This transformation step is crucial because language models process continuous sequences of tokens, not structured data objects. The JSON representation used for storage must be converted into a linear text format that the model can consume. How this conversion is performed directly affects what the model learns about the structure of instructions and responses.

Example prompt formatting:

### Instruction:Explain the concept of overfitting in machine learning. ### Response:Overfitting occurs when a model learns the training data too closely...

In this example, the markers "### Instruction:" and "### Response:" serve as explicit delimiters that tell the model where each section begins. These markers are not arbitrary decoration—they become part of the model's learned vocabulary for understanding task structure.

If the example includes an input field, it might be formatted like this:

### Instruction:Summarize the following paragraph. ### Input:Large language models are trained on massive text datasets and learn statistical patterns in language. These models can generate coherent responses to prompts and perform a wide range of tasks including translation, summarization, and question answering. ### Response:Large language models learn language patterns from large datasets and can perform tasks such as translation, summarization, and question answering.

Notice how the three-field structure naturally accommodates both task-specific instructions and the material those instructions operate on. The instruction defines what operation to perform (summarization), the input provides the content to operate on (the paragraph about language models), and the response demonstrates the expected output.

The choice of formatting markers (like "### Instruction:" and "### Response:") is somewhat arbitrary, but consistency matters. The model learns to associate these markers with different parts of the interaction. Using consistent formatting across all training examples helps the model internalize this structure.

Different projects and research groups have adopted various formatting conventions. Some datasets use alternative formats, such as conversational templates ("User:" and "Assistant:") that mirror chat interfaces. Others employ simpler delimiters like special tokens or line breaks. The ChatML format, for instance, uses XML-like tags to mark different message roles:

<|im_start|>userExplain the concept of overfitting in machine learning.<|im_end|><|im_start|>assistantOverfitting occurs when a model learns the training data too closely...<|im_end|>

This format makes the role boundaries extremely explicit and extends naturally to multi-turn conversations where user and assistant messages alternate.

The key principle remains the same regardless of the specific format chosen: clearly separate the instruction from the response so the model learns which part it should predict. During training, the loss function is computed only on the response tokens, not on the instruction or input tokens. This means the formatting must make it unambiguous where the response begins, because that's where the model's predictions will be evaluated.

The formatting choice also has practical implications for inference. When users interact with the deployed model, their prompts must be formatted in exactly the same way the model saw during training. If the model was trained with "### Instruction:" markers but receives prompts formatted as "User:", it may not perform as expected. This is why model documentation typically includes specific formatting requirements or provides helper functions to ensure consistency.

Another consideration is special token usage. Many modern implementations employ special tokens that mark section boundaries. These tokens are added to the model's vocabulary and serve as unambiguous separators that can't appear in normal text. For example, a format might use <|inst|> to mark the start of an instruction and <|response|> to mark the start of a response. Because these tokens are unique to the formatting system, there's no risk of confusion with similar-looking text in the actual content.

The formatting also determines how the attention mechanism processes the example. In most implementations, the model can attend to all previous tokens when generating each response token. This means it can look back at both the instruction and the input when producing the response. The formatting markers help the model learn to pay attention to the right parts of the context at the right times.

Training Objective

The model is trained to predict the response tokens while treating the instruction and input as context. In technical terms, the loss is computed only on the response portion of the formatted prompt.

To understand this more concretely, consider what happens during a single training step. The model receives the entire formatted prompt—instruction, input (if present), and response—as a sequence of tokens. It processes this sequence from left to right, generating predictions for each position. However, the loss function that drives learning is computed selectively.

When the model processes the instruction and input portions, it generates predictions for the next token at each position, but these predictions are ignored for the purposes of computing loss. The model is not rewarded or penalized based on how well it predicts these tokens. This is because the instruction and input are provided as given context—they represent what the user supplies, not what the model should generate.

Once the model reaches the response section, the loss calculation activates. Now, for each token in the response, the model's prediction is compared against the actual token that should appear. The difference between what the model predicts and what should actually come next determines the loss value. The gradient descent optimization process then adjusts the model's parameters to reduce this loss, making the model slightly better at predicting the correct response tokens.

This selective loss computation is implemented through a technique called loss masking. In practice, this means creating a mask array that marks which positions should contribute to the loss calculation. For instruction and input tokens, the mask value is zero (ignore these positions). For response tokens, the mask value is one (include these in the loss). The training code multiplies the computed loss at each position by the corresponding mask value, effectively zeroing out the loss for non-response tokens.

This means the model is not penalized for failing to predict the instruction or input—it already knows those parts because they're given as context. Instead, all the learning signal comes from how well it predicts the desired response.

The implications of this design choice are profound. By focusing the training signal exclusively on the response, we teach the model a specific skill: given an instruction (and optionally some input material), produce an appropriate output. The model learns to interpret various instruction phrasings, understand what different tasks require, and generate responses that fulfill those requirements.

This is fundamentally different from standard language modeling, where the loss is computed across the entire text sequence. In pure language modeling, the model learns to continue any text it encounters—whether that text is a news article, a conversation, or a random snippet of code. There's no distinction between "context you should understand" and "output you should generate." Everything is treated as text to be predicted.

Instruction tuning, by contrast, explicitly teaches the model to recognize the boundary between input and output. The formatting markers we discussed earlier ("### Instruction:", "### Response:", etc.) help the model learn where this boundary lies. Over thousands or millions of training examples, the model learns that text following the instruction marker should be interpreted as a task specification, while text following the response marker should be generated as task fulfillment.

This targeted training approach is what transforms a general language model into an instruction-following assistant. By repeatedly practicing the pattern of "read instruction → generate appropriate response," the model internalizes the behavior we want: interpreting what the user asks for and producing helpful outputs.

The effectiveness of this approach depends heavily on the diversity and quality of the training examples. If the model sees thousands of examples where instructions ask for summaries and responses provide concise summaries, it learns the general skill of summarization—not just memorization of specific examples, but the underlying pattern of condensing longer text into shorter form while preserving key information. Similarly, exposure to many code generation examples teaches the model to translate natural language descriptions into working code across various programming languages and task types.

The training objective also shapes what the model learns not to do. Because the loss is never computed on the instruction portion, the model doesn't learn to generate instructions spontaneously. This is generally desirable—we want the model to respond to user requests, not invent its own tasks. However, it also means the model's behavior is fundamentally reactive rather than proactive. It waits for instructions rather than taking initiative.

1.1.2 Sources of Instruction Data

Instruction datasets can be created through several distinct approaches, each with its own implications for quality, cost, and scale. Understanding these methods helps explain why modern instruction-tuned models behave the way they do and why dataset construction remains one of the most critical steps in the entire training pipeline.

Human-Created Instruction Data

The most straightforward and historically most reliable method involves having human annotators craft both the instructions and their corresponding responses from scratch. This is the gold standard for data quality, though it comes with significant practical constraints.

This approach was prominently used in the development of InstructGPT, the model that preceded ChatGPT and established many of the instruction-following capabilities we now take for granted. In that project, OpenAI employed professional human labelers who wrote example prompts representing realistic user requests, then wrote ideal responses demonstrating exactly how the model should behave. These labelers were given detailed guidelines about what constitutes a helpful, harmless, and honest response, and their work directly shaped the model's behavior.

The process typically works like this: annotators receive task specifications and quality guidelines, then generate instruction-response pairs covering diverse scenarios. For a customer service use case, they might write instructions like "Help a user reset their password" along with step-by-step responses. For educational applications, they might create instructions asking for explanations of complex concepts, paired with clear, pedagogically sound answers.

Advantages of human-created data:

  • Exceptionally high quality when annotators are skilled and well-guided
  • Clear, natural instructions that reflect real user needs
  • Accurate, helpful, and contextually appropriate responses
  • Captures nuance, cultural context, and domain expertise
  • Allows incorporation of specialized knowledge from subject matter experts

Limitations:

  • Extremely expensive at scale—professional annotators require fair compensation
  • Time-consuming—writing quality examples takes careful thought
  • Difficult to scale beyond tens of thousands of examples without massive resources
  • Requires extensive quality control and annotator training
  • May introduce subtle biases based on annotator demographics and perspectives

Despite these limitations, human-written data remains one of the most valuable sources of instruction examples, particularly for establishing baseline quality standards and for domains requiring expert knowledge. Many organizations use a hybrid approach: human-created examples form a curated core dataset, while other methods expand the dataset size.

The cost factor deserves emphasis. If paying annotators $15-30 per hour and each high-quality instruction-response pair takes 5-10 minutes to create (including thinking time, writing, and revision), the cost per example ranges from $1.25 to $5.00. Creating 100,000 examples could cost $125,000 to $500,000 just in annotation labor, before accounting for management overhead, quality review, and infrastructure.

Synthetic Instruction Data

In recent years, synthetic data generation has emerged as one of the most transformative techniques for scaling instruction datasets. This approach leverages an existing capable language model to generate new training data, creating a virtuous cycle where models help train their successors.

The core insight is elegant: if you already have a model that can follow instructions reasonably well, you can ask it to create new instructions and write responses to them. This automated generation can produce thousands or millions of examples at a fraction of the cost of human annotation.

The Self-Instruct methodology, introduced by researchers at the University of Washington and others, demonstrated how effective this approach can be. The method bootstraps from a small set of manually written seed examples to generate a much larger corpus of instruction data.

The workflow operates in several stages:

  1. Start with a small collection of high-quality seed instructions (typically 100-200 examples written by humans) covering diverse task types.
  2. Prompt a capable language model to generate new instructions that are similar in style and diversity to the seed examples, but different in specific content.
  3. For each generated instruction, prompt the model to produce a corresponding response, effectively asking it to answer its own generated questions.
  4. Apply automated filters to remove low-quality, nonsensical, or problematic outputs.
  5. Optionally, use the generated examples to augment the seed set and repeat the process, allowing the dataset to grow iteratively.

Here's a more complete example showing how this might be implemented in practice:

from openai import OpenAIimport json client = OpenAI() # Seed instructions for bootstrappingseed_instructions = [    "Explain how photosynthesis works in simple terms.",    "Write a Python function to calculate the Fibonacci sequence.",    "Describe three strategies for managing work-related stress."] def generate_new_instruction(seeds):    """Generate a new instruction similar to the seed examples."""    prompt = f"""Below are some example instructions for a language model: {chr(10).join(f"{i+1}. {inst}" for i, inst in enumerate(seeds[:5]))} Generate a new instruction that is different from these examples but similar in style and complexity. The instruction should be clear and specific. New instruction:"""        response = client.chat.completions.create(        model="gpt-5",        messages=[{"role": "user", "content": prompt}],        temperature=0.7    )        return response.choices[0].message.content.strip() def generate_response_for_instruction(instruction):    """Generate a response to the given instruction."""    response = client.chat.completions.create(        model="gpt-4o",        messages=[{"role": "user", "content": instruction}],        temperature=0.7,        max_tokens=500    )        return response.choices[0].message.content.strip() def is_valid_pair(instruction, response):    """Basic quality filter for instruction-response pairs."""    if len(instruction) < 10 or len(response) < 20:        return False    if instruction == response:        return False    if instruction.lower() in response.lower()[:100]:        # Response shouldn't just repeat the instruction        return False    return True # Generate synthetic datasetsynthetic_dataset = [] for i in range(100):    new_instruction = generate_new_instruction(seed_instructions)    new_response = generate_response_for_instruction(new_instruction)        if is_valid_pair(new_instruction, new_response):        synthetic_dataset.append({            "instruction": new_instruction,            "input": "",            "output": new_response        })        print(f"Generated example {len(synthetic_dataset)}") # Save the datasetwith open("synthetic_instructions.json", "w") as f:    json.dump(synthetic_dataset, f, indent=2) print(f"Generated {len(synthetic_dataset)} valid instruction-response pairs")

This code snippet demonstrates a practical implementation of the Self-Instruct methodology described earlier. Let's break down what each part does:

  • Seed instructions: The code starts with a small set of manually written examples (only 3 in this case) that serve as templates for generating new instructions.
  • generatenewinstruction(): This function prompts the language model to create a new instruction similar to the seed examples. It shows the model a few seeds and asks for something different but stylistically consistent.
  • generateresponsefor_instruction(): Once we have a new instruction, this function asks the model to answer its own generated question, creating the response portion of the training pair.
  • isvalidpair(): This quality filter checks for basic problems like overly short content, identical instruction and response, or responses that just repeat the instruction verbatim.
  • Main generation loop: The code generates 100 instruction-response pairs, filtering out invalid ones and saving the results to a JSON file.

Notice how this simple script could generate hundreds or thousands of training examples with minimal human effort—this is the scalability advantage of synthetic data generation. However, as discussed, the output quality depends entirely on the capabilities and biases of the source model (GPT-5 in this example).

This method dramatically increases dataset size while keeping costs manageable. Where human annotation might cost several dollars per example, synthetic generation using API-based models might cost a few cents per example, and using a self-hosted model could reduce costs even further.

The economic implications are substantial. Generating 100,000 instruction-response pairs synthetically might cost $1,000-5,000 in API calls, compared to $125,000-500,000 for human annotation. This cost differential enables experimentation and iteration that would otherwise be prohibitively expensive.

However, synthetic generation introduces a critical risk: synthetic bias or model collapse. If all synthetic data comes from a single model, the resulting dataset inherits that model's weaknesses, stylistic quirks, and knowledge gaps. The student model learns to imitate not just the teacher's strengths but also its limitations.

This phenomenon can be subtle. A model trained primarily on synthetic data might:

  • Reproduce the same response patterns, leading to monotonous or formulaic outputs
  • Perpetuate factual errors present in the synthetic responses
  • Learn overly verbose or artificially formal language patterns
  • Develop similar failure modes to the model that generated the data
  • Lose diversity in reasoning approaches or explanatory styles

Recent research has shown that repeatedly training models on synthetic data from previous model generations can lead to progressive quality degradation—a phenomenon sometimes called "model collapse" or "data incest." Each generation amplifies the biases and artifacts of the previous one, gradually eroding the model's connection to genuine human communication patterns.

To mitigate these risks, practitioners often employ several strategies:

  • Mix synthetic data with human-created examples to maintain grounding in authentic human expression
  • Generate synthetic data from multiple different models to increase diversity
  • Use human reviewers to filter synthetic examples, keeping only the highest quality
  • Continuously introduce fresh human-created seed data to prevent drift
  • Monitor trained models for signs of synthetic bias, such as repetitive phrasing or unusual artifacts

Task-Based Datasets

A third major source of instruction data comes from repurposing existing machine learning benchmarks and task-specific datasets. The NLP research community has created thousands of datasets for specific tasks over the years—question answering, translation, summarization, sentiment analysis, named entity recognition, and many others. These datasets represent enormous collective effort and can be transformed into instruction format with relatively simple processing.

Common task-based datasets that serve as instruction data sources include:

  • Question answering datasets (SQuAD, Natural Questions, TriviaQA)
  • Translation datasets (WMT, OPUS, parallel corpora)
  • Summarization datasets (CNN/Daily Mail, XSum, PubMed)
  • Code generation datasets (APPS, MBPP, HumanEval)
  • Natural language inference datasets (SNLI, MultiNLI)
  • Dialogue datasets (PersonaChat, MultiWOZ)
  • Sentiment and classification datasets (SST, IMDB reviews)

The conversion process involves wrapping the original task data in an instruction-formatted template. Consider a translation dataset that originally contains simple source-target pairs:

Original dataset entry:

English: Hello world French: Bonjour le monde

This can be transformed into instruction format by adding explicit task framing:

Instruction: Translate the following sentence into French.Input: Hello worldResponse: Bonjour le monde

The transformation makes the task structure explicit and teaches the model to respond to natural language instructions rather than expecting a specific input format. Instead of learning "when I see English text, output French text," the model learns "when someone asks me to translate to French, I should produce French text."

For datasets that don't have a natural input-output separation, the conversion requires slightly more creativity. A question-answering dataset might look like:

Instruction: Answer the following question based on the given context.Input: Context: The Eiffel Tower was completed in 1889 for the World's Fair. It stands 330 meters tall and was designed by Gustave Eiffel.Question: When was the Eiffel Tower completed?Response: The Eiffel Tower was completed in 1889.

The instruction provides meta-level framing (what kind of task this is), the input contains the specific materials needed (context and question), and the response demonstrates the expected output format.

Some datasets benefit from instruction variation to improve generalization. Instead of always using "Translate the following sentence," you might rotate through variations:

  • "Translate this English sentence to French:"
  • "How would you say this in French?"
  • "Provide the French translation of:"
  • "Convert the following from English to French:"

This variation teaches the model to recognize different phrasings of the same underlying task, making it more robust to natural variations in how users express their requests.

By transforming existing datasets into instruction format, developers can rapidly generate hundreds of thousands of training examples spanning diverse task types. A single large multi-task collection like FLAN (Fine-tuned Language Net) or P3 (Public Pool of Prompts) might incorporate dozens of underlying datasets, yielding millions of instruction-formatted examples.

The main advantage of this approach is the availability of pre-existing, often well-curated data with known quality characteristics. These datasets have typically undergone peer review, quality control, and validation. The main limitation is that they may not cover the full range of instruction-following behaviors desired in a conversational assistant—they excel at well-defined tasks but may lack examples of creative writing, open-ended dialogue, or nuanced reasoning.

1.1.3 Dataset Diversity

A common mistake in instruction dataset design is focusing too heavily on one task type. When datasets lack diversity, models develop narrow competencies that don't transfer well across domains. This limitation becomes especially apparent in production environments where users make unpredictable requests spanning numerous categories.

For example, if a dataset consists mostly of summarization tasks, the resulting model may excel at condensing articles and documents but struggle with coding, mathematical reasoning, or question answering. The model has learned a specific pattern—"take long text, make it shorter"—but hasn't developed the broader instruction-following capabilities needed for general-purpose assistance.

High-quality instruction datasets intentionally include a wide variety of tasks, ensuring the model encounters diverse reasoning patterns, output formats, and domain knowledge during training. This diversity serves multiple purposes: it prevents overfitting to particular task structures, exposes the model to different styles of human communication, and builds robust capabilities that transfer across domains.

Examples of task categories that should appear in a well-balanced instruction dataset include:

  • Writing explanations: Teaching complex concepts in accessible language, answering "why" and "how" questions, and providing educational content
  • Step-by-step reasoning: Mathematical problem-solving, logical deduction, chain-of-thought demonstrations, and process-oriented tasks
  • Code generation: Writing functions in various programming languages, debugging code, explaining algorithms, and implementing specific functionality
  • Text transformation: Rewriting content for different audiences, changing tone or formality, expanding or condensing text, and format conversions
  • Classification: Categorizing content, sentiment analysis, identifying themes or topics, and labeling data
  • Translation: Converting between natural languages, code-switching, and cross-lingual tasks
  • Dialogue: Multi-turn conversation, context maintenance, roleplay, and interactive assistance
  • Creative writing: Storytelling, poetry, character development, worldbuilding, and imaginative scenarios
  • Data extraction: Pulling structured information from unstructured text, parsing documents, and identifying key facts

Beyond these broad categories, diversity manifests in other important dimensions. Domain diversity ensures coverage across fields like science, history, medicine, law, technology, and arts. Difficulty diversity includes both simple requests (like basic definitions) and complex multi-step problems requiring sophisticated reasoning. Length diversity encompasses both brief queries expecting concise answers and elaborate instructions requiring detailed, nuanced responses.

The distribution across these categories matters significantly. A dataset with 80% coding tasks and 20% everything else will produce a model that feels like a code assistant first and a general assistant second. Research suggests that more uniform distributions—where no single category dominates—tend to produce more balanced capabilities, though the optimal distribution depends on the intended use case.

Some practitioners track task diversity using explicit metadata. Each example might be tagged with its primary task type, domain, difficulty level, and expected response length. This metadata enables systematic analysis: "Do we have enough creative writing examples? Are we underrepresenting scientific reasoning? Is our dataset skewed toward short responses?"

A balanced dataset improves the model's ability to generalize across many domains. When the model encounters a novel request that doesn't perfectly match any training example, it can draw on related skills learned from diverse tasks. A model trained on varied data develops more flexible internal representations—it learns not just specific task patterns but general principles of instruction-following, context interpretation, and appropriate response generation.

This generalization extends to compositional capabilities. A model that has seen separate examples of "write creatively" and "explain technical concepts" can more readily handle requests like "write a creative story that explains quantum mechanics to a child." The diverse training experiences provide building blocks that combine in new ways.

1.1.4 Dataset Cleaning and Filtering

Raw instruction datasets, despite careful initial curation, often contain a variety of issues that can significantly harm model training if left unaddressed. These problems range from obvious errors to subtle inconsistencies that may not be apparent without systematic analysis. Understanding these issues and implementing robust filtering strategies is essential for producing high-quality instruction-tuned models.

The most common problems found in raw instruction datasets include:

  • Incorrect or factually wrong answers: Responses that contain false information, outdated facts, or logical errors. These examples teach the model to produce inaccurate outputs and can be particularly harmful in domains requiring precision, such as medicine, law, or mathematics.
  • Duplicate instructions: The same instruction-response pair appearing multiple times in the dataset. Duplicates cause the model to overfit to specific examples, reducing its ability to generalize. Even near-duplicates—instructions that are only slightly reworded but functionally identical—can create similar problems.
  • Very short or meaningless responses: Answers that are too brief to be helpful, contain only single words when detailed explanations are warranted, or provide vague statements that don't actually address the instruction. These examples teach the model that low-effort responses are acceptable.
  • Harmful or biased outputs: Responses containing offensive language, promoting stereotypes, providing dangerous instructions, or exhibiting systematic biases against particular groups. Even a small percentage of harmful examples can noticeably degrade model behavior.
  • Formatting inconsistencies: Variations in how instructions and responses are structured, inconsistent use of special tokens or delimiters, or mixing of different template formats. These inconsistencies make it harder for the model to learn clean input-output mappings.
  • Instruction-response mismatches: Cases where the response doesn't actually follow the instruction, answers a different question than what was asked, or provides information unrelated to the request.
  • Overly verbose or unnecessarily complex responses: Answers that include excessive preamble, repetitive content, or convoluted explanations when simpler formulations would be clearer and more helpful.

Before training begins, datasets must undergo careful filtering and cleaning to address these issues. The cleaning process typically involves multiple stages, each targeting different types of problems.

Deduplication is usually the first step. Exact duplicates can be identified through simple hashing—converting each example to a unique fingerprint and removing entries with identical fingerprints. Near-duplicates require more sophisticated approaches, such as computing similarity scores between instruction texts and removing pairs above a certain threshold. Some practitioners use MinHash or locality-sensitive hashing (LSH) to efficiently find near-duplicates in large datasets.

Length-based filtering removes responses that are too short to be meaningful or too long to be practically useful. Minimum length thresholds might be set at 20-50 characters for most tasks, though this varies by domain—code generation might require longer responses, while simple classification tasks might legitimately have brief answers. Maximum length filters prevent the inclusion of excessively verbose responses that might teach the model to be unnecessarily wordy.

Quality scoring involves evaluating whether responses actually address their instructions appropriately. Simple heuristics can catch obvious problems: if the instruction and response are identical, something is wrong; if the response contains only punctuation or gibberish, it should be removed; if the instruction asks a question but the response doesn't contain relevant information, it's likely low quality.

Format standardization ensures consistency across the dataset. This might involve converting all examples to use the same template structure, normalizing whitespace and special characters, removing extraneous metadata, and ensuring that multi-turn conversations follow consistent formatting conventions.

A basic filtering pipeline implementing these principles might look like this:

import hashlibfrom collections import defaultdict def compute_hash(text):    """Generate a hash for duplicate detection."""    return hashlib.md5(text.encode('utf-8')).hexdigest() def filter_dataset(samples, min_length=20, max_length=2048):    """    Filter instruction dataset for quality and consistency.        Args:        samples: List of dicts with 'instruction' and 'output' keys        min_length: Minimum response length in characters        max_length: Maximum response length in characters        Returns:        Filtered list of samples    """    cleaned = []    seen_hashes = set()        for sample in samples:        instruction = sample.get("instruction", "").strip()        output = sample.get("output", "").strip()                # Skip empty or malformed samples        if not instruction or not output:            continue                # Check for duplicates        sample_hash = compute_hash(instruction + output)        if sample_hash in seen_hashes:            continue        seen_hashes.add(sample_hash)                # Length filtering        output_length = len(output)        if output_length < min_length or output_length > max_length:            continue                # Check if instruction and output are identical        if instruction == output:            continue                # Filter trivial or low-quality responses        if output.lower() in ["yes", "no", "ok", "done", "n/a"]:            continue                # Check for minimum word count (avoid gibberish)        word_count = len(output.split())        if word_count < 5:            continue                # Standardize formatting        sample["instruction"] = instruction        sample["output"] = output                cleaned.append(sample)        return cleaned # Additional filtering: detect near-duplicates using similaritydef jaccard_similarity(text1, text2):    """Compute Jaccard similarity between two texts."""    words1 = set(text1.lower().split())    words2 = set(text2.lower().split())    intersection = words1.intersection(words2)    union = words1.union(words2)    return len(intersection) / len(union) if union else 0 def remove_near_duplicates(samples, threshold=0.85):    """Remove samples with high instruction similarity."""    filtered = []        for i, sample in enumerate(samples):        is_duplicate = False                for prev_sample in filtered:            similarity = jaccard_similarity(                sample["instruction"],                 prev_sample["instruction"]            )                        if similarity > threshold:                is_duplicate = True                break                if not is_duplicate:            filtered.append(sample)        return filtered

Let's break down what this filtering code does, step by step:

The compute_hash function creates a unique fingerprint for each piece of text. Think of it like generating a Social Security number for each instruction-response pair—if two pairs have the same fingerprint, they're duplicates.

The main filter_dataset function implements several quality checks in sequence:

  • Empty content check: If either the instruction or response is missing or blank, skip it entirely. There's nothing to learn from incomplete examples.
  • Duplicate detection: Using the hash fingerprint, the code tracks which examples it has already seen. If an identical pair appears again, it's discarded. This prevents the model from memorizing specific examples through repeated exposure.
  • Length boundaries: Responses must fall between minimum and maximum character counts (20 and 2048 by default). This removes both unhelpfully brief answers and excessively long, potentially rambling responses.
  • Identity check: If the instruction and output are exactly the same, something has clearly gone wrong in dataset creation. These are removed.
  • Trivial response filtering: Single-word responses like "yes," "no," or "ok" rarely provide meaningful training signal for complex instruction-following, so they're excluded.
  • Word count verification: Responses with fewer than 5 words are likely too sparse to be useful, potentially indicating corrupted or incomplete data.

The jaccard_similarity function measures how similar two texts are by comparing their word overlap. If two instructions share 85% of their words (the default threshold), they're probably asking for the same thing in slightly different words—near-duplicates that should be consolidated.

The removenearduplicates function applies this similarity check across the entire dataset. For each new sample, it compares against all previously accepted samples. If the instruction is too similar to something already included, the duplicate is discarded. This is computationally slower than exact deduplication but catches paraphrased variants that would otherwise slip through.

Together, these functions implement a multi-layered quality gate. Each filter targets a specific type of problem—exact duplicates, near-duplicates, formatting issues, length extremes, and low-effort responses. By applying all these checks systematically, the pipeline transforms a raw dataset that might contain thousands of flawed examples into a clean training set where each example teaches the model something valuable and distinct.

While rule-based filters catch many obvious problems, more sophisticated approaches employ LLM-based quality evaluation. In this approach, a separate language model—often a capable instruction-following model like GPT-4 or Claude—scores each example for qualities like helpfulness, correctness, coherence, and safety. The evaluator model receives the instruction and response, then assigns numerical scores or provides binary judgments about whether the example should be retained.

For instance, the evaluator might be prompted: "Rate the following response on a scale of 1-5 for helpfulness and accuracy. Consider whether the response directly addresses the instruction, provides correct information, and maintains appropriate tone." Examples scoring below a certain threshold are filtered out, while high-scoring examples are retained. This automated quality assessment allows filtering at scale while applying more nuanced judgment than simple heuristics can provide.

Some organizations implement multi-stage filtering pipelines that combine rule-based and model-based approaches. An initial rule-based pass removes obvious problems efficiently, then model-based evaluation provides finer-grained quality assessment on the remaining examples. This hybrid approach balances computational cost with filtering quality—simple rules handle the easy cases, while expensive model evaluations focus on ambiguous examples that require sophisticated judgment.

1.1.5 Manual Review and Expert Curation

Even with automation, manual review remains essential. While automated filtering catches many systematic problems—duplicates, formatting errors, length violations—it cannot reliably assess nuanced qualities like factual accuracy in specialized domains, appropriateness of tone, or subtle safety concerns that require human judgment.

Expert reviewers bring domain knowledge and contextual understanding that automated systems lack. A filter can detect that a medical response is properly formatted and uses relevant terminology, but only a medical professional can verify whether the advice is actually correct and safe. Similarly, automated systems might flag obvious toxicity, but human reviewers can identify subtle biases, culturally insensitive framings, or responses that are technically accurate but pedagogically unhelpful.

The manual review process typically focuses on several key dimensions:

  • Factual accuracy: Reviewers verify that responses contain correct information, especially in domains where errors could cause real harm. For scientific, medical, legal, or financial instructions, subject matter experts check that the model's responses align with current knowledge and best practices. They flag outdated information, common misconceptions, or subtle errors that might seem plausible but are actually incorrect.
  • Clarity and coherence: Responses should be well-structured and easy to understand. Reviewers identify examples where the model's output is confusing, uses unnecessary jargon without explanation, or fails to organize information logically. Clear communication is particularly important for educational instructions, where the goal is to help users learn.
  • Instruction-response alignment: The response must actually address what the instruction asks for. Reviewers catch cases where the model provides related but off-topic information, answers a different question than what was asked, or includes extraneous content that dilutes the core answer. This alignment check is especially important for multi-part instructions where the response should address each component.
  • Safety and ethical compliance: Human reviewers evaluate whether responses could cause harm, promote dangerous activities, contain offensive content, or exhibit biases against particular groups. This includes both obvious violations—instructions for illegal activities, hate speech—and subtle issues like consistently portraying certain professions with gender stereotypes or providing advice that could be harmful in certain contexts.
  • Tone and style appropriateness: Different instructions call for different communication styles. Reviewers ensure that formal requests receive appropriately professional responses, that creative prompts yield engaging outputs, and that sensitive topics are handled with care. A response explaining a difficult concept to a child should sound very different from a technical explanation for experts.

In practice, organizations implement manual review at different scales depending on resources and requirements. Some teams manually review every single training example, though this is only feasible for smaller datasets of a few thousand samples. More commonly, reviewers examine a representative sample—perhaps 5-10% of the dataset—using stratified sampling to ensure coverage across different instruction types, difficulty levels, and domains.

When full manual review isn't feasible, teams often prioritize reviewing high-impact categories: examples in sensitive domains like healthcare or legal advice, responses to potentially harmful instructions, and examples that automated filters flagged as borderline cases. This targeted approach focuses human expertise where it provides the most value.

Reviewers typically work with structured rubrics that define clear criteria for each quality dimension. Rather than making purely subjective judgments, they score examples on specific scales—rating factual accuracy from 1-5, checking boxes for specific safety concerns, or categorizing tone appropriateness. This structured approach improves consistency across reviewers and creates actionable feedback for dataset improvement.

In large-scale projects, datasets may go through several review cycles before training begins. An initial round identifies major problems and establishes clearer quality standards. After filtering and corrections based on first-round feedback, a second review pass checks that improvements were implemented correctly and catches remaining edge cases. Some organizations even conduct post-training review, where reviewers examine model outputs on held-out examples to verify that the training data successfully taught intended behaviors.

The insights from manual review often feed back into improving automated filtering. If reviewers consistently flag a particular type of problem—say, responses that cite nonexistent sources—developers can create new automated checks to detect similar issues throughout the dataset. This creates a virtuous cycle where human expertise scales through automation, while automated systems free reviewers to focus on cases requiring sophisticated judgment.

1.1.6 Instruction Dataset Size

Instruction datasets vary considerably in size, and understanding the relationship between dataset scale and model performance is crucial for anyone building instruction-following systems. The evolution of instruction tuning reveals an interesting trajectory: early pioneering work operated with remarkably small datasets, while modern approaches leverage massive collections of examples.

The earliest instruction-tuning experiments, such as those with FLAN (Fine-tuned Language Net) and T0, used datasets containing just a few thousand carefully constructed examples. These pioneering efforts demonstrated that even modest amounts of instruction data could dramatically improve a model's ability to follow directions. Researchers hand-crafted templates for common tasks like sentiment analysis, question answering, and text summarization, then generated examples by filling these templates with diverse content. Despite their limited scale, these datasets proved that instruction tuning was a viable approach to steering model behavior.

As the field matured, dataset sizes grew exponentially. Modern instruction datasets often contain hundreds of thousands of examples—sometimes crossing into the millions. Projects like Alpaca generated 52,000 instruction-following examples using GPT-3.5. Databricks' Dolly dataset contributed 15,000 human-generated instruction-response pairs. The OpenAssistant project collected over 160,000 conversations through crowdsourcing. More recently, synthetic datasets generated by powerful models have scaled to millions of examples, covering increasingly diverse tasks and domains.

However, the relationship between dataset size and model quality is not simply linear. Size alone does not guarantee superior performance, and this is a critical point that practitioners must internalize. A smaller but meticulously curated dataset—where every example has been verified for accuracy, clarity, and alignment—can sometimes outperform a much larger collection riddled with noise, duplicates, and low-quality responses.

Why does quality trump quantity in many cases? First, models learn most effectively from clear, consistent examples. If a dataset contains contradictory examples—say, one response saying a task is impossible while another shows how to accomplish it—the model receives mixed signals that dilute the training effect. Second, duplicated or near-duplicate examples don't provide new information; they simply cause the model to memorize specific patterns rather than learning generalizable instruction-following behavior. Third, low-quality responses teach bad habits. If 30% of your dataset contains responses that are partially incorrect, poorly structured, or off-topic, you're actively training the model to produce similar flawed outputs.

The diminishing returns of dataset size become apparent when comparing different approaches. A dataset of 10,000 expertly curated, diverse examples—covering a wide range of task types, difficulty levels, and domains—might produce a more capable model than 100,000 examples scraped indiscriminately from the internet without quality filtering. The former teaches the model a broad set of skills through clear examples; the latter teaches the model to imitate the average quality of internet text, which includes plenty of mediocrity and errors.

This quality-versus-quantity trade-off has important practical implications for resource allocation. If you have a limited budget for dataset creation, you face a choice: hire 100 annotators to quickly generate 50,000 examples with minimal review, or hire 20 expert annotators to carefully craft and verify 10,000 examples. The research increasingly suggests that the latter approach often yields better results, particularly when those 10,000 examples are strategically selected to cover important capabilities.

Dataset diversity also plays a crucial role in determining effective size. A dataset with 50,000 examples might seem large, but if 40,000 of those examples are all creative writing prompts and only 10,000 cover other tasks, the model will become specialized in creative writing at the expense of other capabilities. Conversely, a dataset of 20,000 examples that includes balanced representation across mathematical reasoning, coding, factual question-answering, creative tasks, and conversational interactions provides exposure to a broader range of behaviors. The effective training signal comes not just from total volume but from coverage across the capability space you want your model to inhabit.

Recent research has explored the concept of "skill diversity" in instruction datasets. Rather than measuring dataset size purely by example count, researchers analyze how many distinct skills or task types the dataset covers. A dataset might contain many examples of addition problems, but they all teach the same underlying skill. Meanwhile, a dataset with fewer total examples but greater task diversity—covering addition, subtraction, word problems, equation solving, and mathematical proof—teaches a richer set of capabilities. This perspective suggests that dataset curation should prioritize covering the skill space comprehensively rather than accumulating large numbers of examples within narrow task categories.

The goal, then, is not simply to maximize the raw number of samples. Instead, practitioners should aim to build datasets that represent the full spectrum of tasks and behaviors they want their models to learn. This means actively identifying gaps in task coverage, ensuring representation across difficulty levels, including examples that demonstrate important nuances like handling ambiguous instructions or explaining reasoning steps, and maintaining consistent quality standards throughout the dataset.

In practice, many successful instruction-tuning projects follow a hybrid approach: they start with a large-scale dataset to provide broad coverage, then apply aggressive filtering to remove low-quality examples, and finally augment the filtered dataset with hand-curated examples in areas where automated collection produces poor results. This combines the efficiency of large-scale data collection with the quality assurance of human curation.

As we move forward in understanding instruction tuning, keep in mind that the number of examples matters, but what those examples teach matters more. A well-designed dataset of moderate size, with careful attention to quality, diversity, and coverage, forms a stronger foundation for instruction-following behavior than a massive but poorly curated collection.