Step 4: Clean + split → JSONL (training format)

Section 5 of 10-~ 7 min read-Synced from Cuantum content

Now that you have a raw dataset, you need to transform it into the format that the training pipeline expects. This step involves three things: cleaning your data to remove inconsistencies, splitting it into training and evaluation sets, and converting it to JSONL format with the proper prompt structure.

This might seem like busywork, but it's not. Data preprocessing is where most real-world ML projects succeed or fail. A clean, well-structured dataset will train faster, generalize better, and produce more reliable outputs. A messy dataset will cause all kinds of subtle problems that are hard to debug later.

We're going to write a script that does all three steps in one pass. It will read your raw.json file, clean each example, split the data into train and eval sets, and write out two JSONL files that are ready to be loaded by the trainer.

Create scripts/make_jsonl.py:

import jsonimport random random.seed(42) def clean_text(s: str) -> str:    return " ".join((s or "").split()).strip() def to_prompt(ex):    # Simple, readable training format    inst = clean_text(ex["instruction"])    inp = clean_text(ex.get("input", ""))    out = clean_text(ex["output"])     prompt = f"### Instruction:\n{inst}\n"    if inp:        prompt += f"### Input:\n{inp}\n"    prompt += "### Response:\n"    return prompt, out def main():    with open("data/raw.json", "r", encoding="utf-8") as f:        data = json.load(f)     cleaned = []    for ex in data:        if not ex.get("instruction") or not ex.get("output"):            continue        ex["instruction"] = clean_text(ex["instruction"])        ex["input"] = clean_text(ex.get("input", ""))        ex["output"] = clean_text(ex["output"])        if len(ex["output"].split()) < 3:            continue        cleaned.append(ex)     # Save cleaned    with open("data/cleaned.json", "w", encoding="utf-8") as f:        json.dump(cleaned, f, indent=2, ensure_ascii=False)     # Split train/eval    random.shuffle(cleaned)    n = len(cleaned)    eval_size = max(1, int(0.1 * n))    eval_set = cleaned[:eval_size]    train_set = cleaned[eval_size:]     def write_jsonl(path, rows):        with open(path, "w", encoding="utf-8") as f:            for ex in rows:                prompt, answer = to_prompt(ex)                # TRL SFTTrainer can train on a single "text" field                # where the target is included in the same sequence.                record = {"text": prompt + answer}                f.write(json.dumps(record, ensure_ascii=False) + "\n")     write_jsonl("data/train.jsonl", train_set)    write_jsonl("data/eval.jsonl", eval_set)     print(f"Cleaned: {len(cleaned)} | Train: {len(train_set)} | Eval: {len(eval_set)}") if __name__ == "__main__":    main()

Let's walk through what this script does, step by step.

The clean_text function normalizes whitespace. It removes extra spaces, newlines, and tabs, and strips leading/trailing whitespace. This ensures that your examples don't have weird formatting artifacts that might confuse the tokenizer or make the outputs inconsistent.

The to_prompt function converts each example into the prompt format that the model will see during training. This is where you define the structure: ### Instruction:, optionally ### Input:, and ### Response:. The model will learn to recognize this structure and generate responses that follow it. Notice that the function returns both the prompt (without the answer) and the output (the answer itself). We'll combine them later.

The main function does the heavy lifting. First, it loads your raw data and filters out any examples that are missing an instruction or output. Then it cleans each field using clean_text. It also filters out examples where the output is fewer than three words—these are usually too short to be useful and can cause the model to learn bad habits like generating one-word responses.

After cleaning, the script saves a copy of the cleaned data to data/cleaned.json. This is optional, but it's useful for debugging. If something goes wrong later, you can inspect this file to see what your data looked like after cleaning but before conversion to JSONL.

Next, the script shuffles the data and splits it into training and evaluation sets. The evaluation set is 10% of the total data, with a minimum of 1 example. Shuffling is important because it ensures that your eval set isn't just the last 10% of examples you wrote—it's a random sample, which gives you a better sense of how well the model generalizes.

Finally, the write_jsonl helper function writes out the train and eval sets in JSONL format. Each line is a JSON object with a single text field that contains the full training example: the prompt and the answer, concatenated together. This is the format that SFTTrainer expects. The trainer will tokenize the full sequence and use causal language modeling to teach the model to predict the answer given the prompt.

Run it:

python scripts/make_jsonl.py

You should see output like this:

Cleaned: 103 | Train: 92 | Eval: 11

You now have two files in your data/ folder:

data/train.jsonl — your training set
data/eval.jsonl — your evaluation set

These files are ready to be loaded by the trainer. Each line is a standalone training example, formatted exactly how the model will see it during training. If you open one of these files, you'll see lines that look like this:

{"text": "### Instruction:\nExplain gradient accumulation in simple terms.\n### Response:\nGradient accumulation lets you train with a small batch size by adding gradients over several steps before updating the model. It's like pretending you used a bigger batch without needing more GPU memory."}

This is the raw input the model will train on. It will learn to predict each token in the response, given the instruction that comes before it. Over many examples, it will learn the pattern: when it sees ### Instruction: followed by a task, it should generate a response that starts after ### Response: and matches the style and content of your training data.

One more thing: notice that we're using a 90/10 train/eval split. This is a reasonable default for small datasets. If you have fewer than 50 examples, you might want to increase the eval size to 20% so you have enough data to measure generalization. If you have thousands of examples, you can reduce it to 5% or even 2%. The goal is to have enough eval examples to get a reliable signal, but not so many that you're wasting training data.