Step 9: Save, Version, and Reuse Your Aligned Chatbot

Section 10 of 19-~ 5 min read-Synced from Cuantum content

Once your DPO training completes and you've validated the aligned model's behavior through systematic evaluation, you have a deployable checkpoint that behaves differently—and hopefully better—than your base model. This checkpoint can be loaded, served, and used just like any other fine-tuned model, but it now reflects the preferences you've encoded through your chosen vs. rejected pairs.

Saving and Versioning Your Aligned Model

Model versioning is critical when doing iterative alignment work. Each training run represents a hypothesis about what preference data will produce the desired behavior, and you need to be able to compare versions systematically to know whether you're improving.

Recommended versioning structure:

Organize your model outputs in a clear directory structure that makes it easy to track different iterations:

outputs/ch3_dpo_chatbot/v1_baseline — your first training run with initial preference data
outputs/ch3_dpo_chatbot/v2_safety_focused — second iteration after adding safety-focused preference pairs
outputs/ch3_dpo_chatbot/v3_conciseness — third iteration targeting verbose responses

Each version directory should contain not just the model checkpoint, but also comprehensive metadata that makes the training reproducible and interpretable.

Essential metadata to store with each version:

The rubric used for preference judgments: Save the exact criteria (helpfulness, safety, honesty, conciseness, etc.) and how they were weighted or prioritized. If your rubric evolves between versions, this documentation shows exactly what changed.
Beta value: Record the DPO beta hyperparameter used in training. This controls how strongly the model is pushed toward chosen responses vs. rejected ones. If you compare two versions with different betas, you need to know which difference came from data vs. hyperparameters.
Dataset size and composition: Log the total number of preference pairs, and ideally a breakdown by category (e.g., "150 total pairs: 60 safety, 40 factual accuracy, 30 conciseness, 20 edge cases"). This helps you understand coverage gaps when diagnosing failures.
Sampling settings for candidate generation: If you generated candidates with specific temperature, top-p, or other sampling parameters, record those. Different sampling strategies produce different quality distributions of chosen/rejected pairs, and you may want to reproduce or adjust this in future iterations.
Training hyperparameters: Learning rate, number of epochs, batch size, optimizer settings—everything needed to exactly reproduce the training run.
Evaluation prompts and results: Store both your fixed evaluation set and the model's outputs on those prompts for this version. This creates a historical record you can compare across versions: "v1 hallucinated on 4/10 technical questions, v2 on 2/10, v3 on 1/10."
Training date and duration: Practical metadata that helps you remember context ("this was the version we trained right after discovering the hallucination issue").

A simple way to store this metadata is in a metadata.json file alongside each model checkpoint:

{  "version": "v2_safety_focused",  "date": "2026-03-02",  "base_model": "HuggingFaceH4/mistral-7b-sft-beta",  "dataset": {    "total_pairs": 175,    "breakdown": {      "safety": 70,      "factual_accuracy": 45,      "conciseness": 35,      "edge_cases": 25    },    "dataset_file": "preferences_v2.jsonl"  },  "training": {    "beta": 0.1,    "learning_rate": 5e-7,    "epochs": 3,    "batch_size": 4,    "optimizer": "adamw"  },  "candidate_generation": {    "temperature": 0.9,    "top_p": 0.95,    "candidates_per_prompt": 4  },  "rubric": {    "dimensions": ["safety", "helpfulness", "honesty", "conciseness"],    "priority": "safety > honesty > helpfulness > conciseness"  },  "evaluation": {    "fixed_test_prompts": 15,    "stress_test_prompts": 10,    "results_file": "evaluation_v2_results.json"  },  "notes": "Added 25 new safety pairs after v1 failed to refuse harmful requests politely. Improved refusal behavior on 6/7 safety test prompts."}

This metadata turns each model version into a complete, interpretable artifact. If you come back to this project in six months, or if a teammate needs to understand your alignment decisions, they can read exactly what you did and why.