Step 5: Build a Grounded Evaluation Set for Hallucination Detection

If you want MT-Bench-style scoring, you can add a judge model to rate each transcript. This transforms your raw conversational transcripts into quantified quality assessments, making it easier to compare model versions at scale. While manual inspection of transcripts reveals the what (what the model actually said), judge-based scoring reveals the how much (how much better or worse one model is than another across your entire test set).

The judge model approach works by treating another language model—typically a larger, more capable one—as an automated evaluator. You feed it the conversation transcript along with evaluation criteria, and it produces a structured assessment. This is the same pattern used in the original MT-Bench paper, where GPT-4 served as the judge for evaluating other models' conversational abilities. The judge isn't perfect (it has its own biases and blind spots), but it's consistent, fast, and often correlates well with human judgment.

A practical approach:

• Use AI-as-a-judge: Send each conversation transcript to a capable model (GPT-4, Claude, or even a strong open-source model like Llama-3-70B) along with a detailed rubric. The judge model reads the entire conversation, evaluates how well the assistant maintained coherence, followed instructions, provided helpful responses, and avoided errors. This mimics what a human evaluator would do, but at machine speed and cost.
• Force a JSON score from 1–10: Structure your judge prompt to require a specific output format—typically a JSON object containing numeric scores and brief justifications. The 1-10 scale provides enough granularity to distinguish between clearly bad (1-3), mediocre (4-6), good (7-8), and excellent (9-10) conversations, while avoiding the false precision of continuous scores. By forcing JSON output, you make parsing deterministic—no need to write fragile regex patterns to extract scores from freeform text. The structured format also encourages the judge to be systematic rather than rambling.
• Use the same rubric every time: Consistency is more important than perfection in judge design. Your rubric should specify exactly what you're measuring: instruction following, conversational coherence, helpfulness, factual accuracy, appropriate refusals. Include these criteria in your judge prompt verbatim for every evaluation. This repetition ensures that a score of 7 means the same thing whether you're evaluating conversation 1 or conversation 20, and whether you ran the evaluation last week or next month. Rubric drift—where your standards unconsciously shift over time—is a common source of unreliable evaluation results.

You can extend this later for more rigorous comparisons. The basic judge-based scoring gives you aggregate quality metrics that are expensive to produce manually but cheap to compute automatically. Once that's working, you can add multi-aspect scoring (separate scores for helpfulness, safety, factuality), pairwise comparisons (which response is better rather than absolute scoring), or even ensembles of multiple judge models to reduce individual judge bias. But start simple: one judge, one rubric, consistent application. That alone will reveal whether your fine-tuning improved conversational quality or degraded it, which is the question you actually need answered before investing in more sophisticated evaluation infrastructure.