Step 2: Build an MT-Bench-Style Multi-Turn Test Set

Section 3 of 11-~ 3 min read-Synced from Cuantum content

You need two models to run this evaluation workflow. This comparison is the foundation of the entire project—without it, you have no baseline to measure improvement against.

Base model: This is your original checkpoint before any fine-tuning. It represents the model's capabilities in its pre-trained state, before you applied SFT, LoRA, DPO, or any other alignment technique. Think of this as your control group in an experiment. Every claim you make about fine-tuning improving performance is really a claim about the delta between this model and your tuned version.

Fine-tuned model: This is whatever you trained in earlier chapters—your SFT model, your LoRA adapter merged back into the base, or your DPO-aligned checkpoint. This represents your hypothesis about what training should accomplish. Maybe you fine-tuned on customer support dialogues and expect better helpfulness. Maybe you applied DPO to reduce harmful outputs. Whatever your goal was, this model embodies the result of that training process.

The comparison between these two models reveals the actual effects of your training intervention. Sometimes those effects align with your intentions. Sometimes they don't. The only way to know is to measure both models on identical tasks under identical conditions.

Example choices:

Base: TinyLlama/TinyLlama-1.1B-Chat-v1.0
Fine-tuned: outputs/ch3_dpo_chatbot/final (or your Chapter 1/2 outputs)

If you're using a different model family—say, a Llama 2 or Mistral variant—just substitute the appropriate Hugging Face model ID for the base. The key requirement is that your fine-tuned model must be derived from that same base architecture. You can't compare apples to oranges and expect meaningful insights about what your training accomplished.

If you applied LoRA or another parameter-efficient method, make sure your fine-tuned model path points to either the merged checkpoint or includes the adapter loading logic. The evaluation scripts assume you can load both models the same way through Hugging Face's AutoModelForCausalLM. If your setup requires custom loading (say, you're using a quantized model or a non-standard adapter), you'll need to modify the load_model() function accordingly.

For the code below, set these constants at the top of each script:

BASE_MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"TUNED_MODEL = "outputs/your_finetuned_model"

Replace "outputs/your_finetuned_model" with the actual path to your trained checkpoint. If you're loading from Hugging Face Hub instead of a local directory, use the repository ID (e.g., "your-username/your-model-name").

These constants will be used by both evaluation scripts, so maintaining consistency here ensures you're always comparing the same model pair across all tests. If you later train a new version and want to evaluate it, you simply update TUNED_MODEL and re-run the scripts. The evaluation infrastructure stays the same; only the model being measured changes.