Step 6: Quick evaluation (does it actually follow your instructions?)

Section 7 of 10-~ 7 min read-Synced from Cuantum content

Now that training is complete, it's time to answer the most important question: did it actually work? Not "did the loss go down"—that's just a number on a screen. The real question is: does your model now behave differently? Does it follow instructions in the style you trained it on?

This step is about building intuition. You're going to run a direct comparison between the base model (before fine-tuning) and your fine-tuned checkpoint (after training). This "before and after" test is one of the most powerful evaluation tools you have, especially early in a project when you're still figuring out whether your approach is working at all.

We'll keep this simple and practical: load both models, feed them the same prompt, and see what they produce. No complex metrics yet—just your eyes and your judgment.

Setting up the comparison script

Create a new file called scripts/inference_test.py. This script will load both the base model and your fine-tuned model, then generate responses to the same prompt so you can compare them side by side.

from transformers import AutoTokenizer, AutoModelForCausalLMimport torch BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"FT = "outputs/ch1_sft_tinyllama/final" def generate(model, tokenizer, prompt, max_new_tokens=120):    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)    with torch.no_grad():        out = model.generate(            **inputs,            max_new_tokens=max_new_tokens,            do_sample=True,            temperature=0.7,            top_p=0.9        )    return tokenizer.decode(out[0], skip_special_tokens=True) def main():    prompt = (        "### Instruction:\n"        "Explain gradient accumulation in simple terms.\n"        "### Response:\n"    )     base_tok = AutoTokenizer.from_pretrained(BASE, use_fast=True)    if base_tok.pad_token is None:        base_tok.pad_token = base_tok.eos_token    base_model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")     ft_tok = AutoTokenizer.from_pretrained(FT, use_fast=True)    if ft_tok.pad_token is None:        ft_tok.pad_token = ft_tok.eos_token    ft_model = AutoModelForCausalLM.from_pretrained(FT, device_map="auto")     print("\n--- BASE MODEL ---")    print(generate(base_model, base_tok, prompt))     print("\n--- FINE-TUNED MODEL ---")    print(generate(ft_model, ft_tok, prompt)) if __name__ == "__main__":    main()

Let's walk through what this script does:

Model paths: We define two constants at the top: BASE points to the original TinyLlama model on Hugging Face, and FT points to your fine-tuned checkpoint that you just saved in the previous step.

Generation function: The generate() function handles the actual text generation. It takes a model, tokenizer, and prompt, then returns the generated text. We're using torch.no_grad() to disable gradient computation (since we're only doing inference, not training), which saves memory. The generation parameters are set to produce reasonably creative but coherent outputs: temperature=0.7 adds some randomness without making the output too wild, and top_p=0.9 uses nucleus sampling to keep the model from choosing extremely unlikely tokens.

Prompt format: Notice that we're using the exact same instruction format that we used during training: "### Instruction:" followed by the instruction text, then "### Response:". This is critical. If you trained the model on a specific format but test it with a different format, the model won't know how to respond properly. Always match your inference prompts to your training format.

Loading both models: In the main() function, we load both the base model and the fine-tuned model into memory. This does require enough VRAM to hold both models simultaneously (for TinyLlama at 1.1B parameters, this should work on most GPUs with 8GB+ VRAM). If you're running low on memory, you can modify the script to load them one at a time instead.

Side-by-side comparison: We generate from both models using the same prompt and print the results with clear labels. This makes it easy to see the difference at a glance.

Running the test

Execute the script:

python scripts/inference_test.py

The first time you run this, it will download the base model from Hugging Face (if you haven't already), then load both models and generate outputs. This might take 30 seconds to a minute depending on your hardware.

What you're looking for

When you compare the two outputs, here's what you want to see:

The fine-tuned model should sound more like your dataset outputs — If you trained on examples that are concise and structured, the fine-tuned model should produce concise and structured responses. If you trained on examples that use specific terminology or phrasing, you should see that reflected in the output. The base model, by contrast, will sound more generic and may use different wording or structure.
It should follow your formatting and tone more consistently — Does your dataset use bullet points? Short sentences? A particular level of formality? The fine-tuned model should mirror those patterns more closely than the base model. This is one of the most visible signs of successful fine-tuning: the model has learned not just what to say, but how to say it.
It should be less "generic" — Base models are trained on massive, diverse datasets, which makes them versatile but often bland. Fine-tuning on a focused dataset should make the model more opinionated, more consistent, and more aligned with the specific style you're targeting. If the fine-tuned output still sounds exactly like the base model, that's a sign that either your dataset wasn't distinctive enough, or the training didn't converge properly.

Don't expect perfection on the first try. What you're looking for is movement in the right direction. Even a subtle shift toward your desired style is a success at this stage—it means the training loop is working, and you can now iterate on the dataset to improve quality.

Red flags to watch for

Sometimes the fine-tuned model will perform worse than the base model. Here are common warning signs:

The model repeats the prompt verbatim — This usually means the training examples didn't have a clear enough separation between instruction and response, or the model didn't see enough diversity in response styles.
The output is incoherent or repetitive — This can happen if you overtrained (too many epochs on too small a dataset) or if your training examples were inconsistent or low-quality.
The model ignores the instruction entirely — This often points to a format mismatch between training and inference, or it means the model hasn't learned to associate the instruction format with the expected behavior.

If you see any of these issues, don't panic—this is all part of the process. The troubleshooting section later in this chapter will help you diagnose and fix these problems.