Project Overview and Files

In this project, you will build a practical evaluation workflow that answers two important questions:

• Does your fine-tuned (or aligned) model perform better than the base model in multi-turn conversations?
• Does it hallucinate less, especially when answers must be grounded in provided context?

These questions matter because fine-tuning changes model behavior in ways that aren't always obvious from loss curves alone. A model can achieve lower training loss while becoming worse at conversation flow, or it might become more accurate on average while developing subtle hallucination patterns that only appear in specific contexts. Without systematic evaluation, you're flying blind.

You will implement:

• An MT-Bench-style multi-turn evaluation harness (lightweight and reproducible)
• A small hallucination-detection pipeline for grounded answering

The MT-Bench component focuses on conversational coherence across multiple turns. Real conversations aren't single-shot Q&A—they involve context retention, follow-up questions, corrections, and changing constraints. Your harness will test whether your model can track these dynamics or whether it loses the thread after the first response.

The hallucination pipeline addresses a different failure mode: making up information when asked to ground answers in specific context. This is critical for applications like customer support, documentation assistance, or any scenario where "I don't know" is better than a confident wrong answer.

You will compare:

• A base model (your starting checkpoint)
• A fine-tuned model (SFT, PEFT, or DPO-aligned)

This comparison reveals what your training actually changed. Sometimes fine-tuning fixes one problem while creating another. Maybe your model becomes more helpful but also more prone to making things up. Maybe it becomes more cautious but overly rigid. The only way to know is to measure both models on the same tasks.

By the end, you will have a reusable evaluation script you can run every time you train a new model version.

This is key: evaluation infrastructure should be built once and used many times. Every experiment you run should flow through the same pipeline. That way, you can track whether changes are actually improvements, catch regressions early, and build institutional knowledge about what training strategies work for your specific use case.

Deliverables (what you will produce)

• data/mtbench_conversations.json: your multi-turn conversation set
• evaluate_mtbench.py: runs both models on the MT-Bench conversations
• outputs/mtbench_results.json: saved transcripts plus basic signals (for example, refusal counts)
• data/grounded_eval.json: your grounding evaluation set
• evaluate_grounding.py: runs context-only QA and flags unsupported claims
• outputs/hallucination_results.json: per-item flags and summary metrics

Success criteria (minimum bar)

• Both scripts run end-to-end without manual intervention.
• Output files are readable and make base vs tuned comparison straightforward.
• You can identify at least one concrete improvement in the tuned model, supported by outputs (for example, fewer multi-turn breakdowns, or lower unsupported-sentence rate).

Success criteria (strong outcome)

• The direction of the improvement is stable across at least two runs (different random seeds).
• You can point to specific conversation IDs and grounded item IDs that explain why the tuned model is better.