Step 8: Common issues and how to fix them

Problem: The model repeats the prompt or rambles

Fixes:

This is one of the most common issues in instruction tuning, especially with smaller models. The model may simply echo back parts of the instruction, or it may generate verbose, circular responses that don't add value.

• Improve your dataset outputs (make them crisp and consistent)Review your training data carefully. If your example outputs are wordy, meandering, or inconsistent in tone, the model will learn to mimic that behavior. Edit your dataset so every response is direct, concise, and on-point. Think of each output as a "gold standard" example of how you want the model to behave.
• Reduce temperature during inference (temperature=0.3)Temperature controls randomness in generation. A high temperature (like 0.9 or 1.0) makes the model more creative but also more prone to wandering off-topic or repeating itself. Lowering it to 0.3 or even 0.1 makes the model more deterministic and focused, which often reduces rambling.
• Add more examples where the ideal response is short and directIf your dataset is full of long-winded answers, the model will learn to be long-winded. Balance it out by including examples where the instruction asks for a brief answer and the response is genuinely brief. This teaches the model that conciseness is sometimes the right choice.

Problem: The model ignores the "Instruction/Response" formatting

Fixes:

Sometimes the model generates responses that don't follow the structure you've set up (like "### Instruction" and "### Response"). This usually means the model hasn't learned to associate that format with the desired behavior.

• Ensure every training sample uses the same templateConsistency is key. If even 10% of your training examples use a different format (like "Question:" instead of "### Instruction:"), the model will get confused. Standardize your entire dataset to use one clear, repeatable template.
• Avoid mixing multiple formats in the same dataset early onIt's tempting to throw in examples from different sources (e.g., some formatted as chat, some as Q&A, some as instruction-response). But during your first fine-tuning run, stick to one format. Once the model reliably follows that format, you can experiment with multi-format datasets in later iterations.

Problem: Training loss decreases but outputs don't improve much

Fixes:

This is a frustrating situation: the loss curve looks great, but when you actually test the model, it doesn't seem any better. This usually points to a data quality or data-prompt mismatch issue.

• Your dataset may be too small or too repetitiveIf you only have 50 examples and they're all very similar, the model will overfit to those specific examples without learning the broader pattern you want. Aim for at least 100–500 diverse examples for a first pass, and make sure they cover different phrasings, tones, and levels of difficulty.
• Add diversity: different wording, different difficulty, more real-world inputsDon't just paraphrase the same instruction 100 times. Include a range of scenarios: some easy, some hard, some formal, some casual. The more varied your training set, the more robust your model will be when faced with new prompts.
• Validate your prompts match your training formatOne common mistake: you train on prompts that look like "### Instruction: Do X\n### Response:\n", but then you test with prompts that look like "Do X". The model won't know how to respond because it never saw that format during training. Always test with the exact same prompt structure you used in training.