1.4 What Could Go Wrong?
Supervised Fine-Tuning may appear straightforward on the surface: collect instruction–response pairs, preprocess them, and train the model. In practice, however, many things can go wrong during this stage. Some problems are obvious, such as training crashes or memory errors. Others are more subtle and may only appear after deployment, when users interact with the model in unpredictable ways.
For this reason, successful instruction tuning requires more than simply running training scripts. It requires careful monitoring, thoughtful dataset design, and an understanding of the common pitfalls that arise when adapting large language models.
In this section, we explore several of the most common issues encountered during instruction tuning and discuss practical strategies for avoiding them.
1.4.1 Poor Dataset Quality
Low-quality instruction data leads to low-quality models.
This may sound obvious, but it is one of the most common causes of disappointing fine-tuning results. If instruction datasets contain unclear prompts, incorrect answers, or inconsistent formatting, the model will learn those patterns.
For example, consider the following training example:
Instruction:
Explain what gradient descent is.
Response:
Gradient descent is a thing used in AI sometimes.
Although the response is not technically incorrect, it is vague and unhelpful. If many training examples resemble this one, the resulting model will produce similarly shallow answers.
Even worse, incorrect responses can cause the model to learn false information.
Example:
Instruction:
What is the capital of Australia?
Response:
Sydney
Because Sydney is not the capital of Australia (the correct answer is Canberra), training on this example reinforces factual errors.
This is why dataset validation and filtering are essential steps before training begins.
Common quality checks include:
- Removing duplicates
- Filtering extremely short responses
- Verifying factual correctness
- Ensuring consistent formatting
- Manually reviewing samples
Some pipelines also use a second LLM to evaluate the quality of responses.
Example pseudo-code:
def evaluate_response(instruction, response): prompt = f"""Evaluate the following response for correctness and clarity. Instruction: {instruction}Response: {response} Score from 1 to 5."""While automated checks help, human review remains one of the most reliable methods for maintaining dataset quality.
1.4.2 Overfitting to Instruction Style
Another common issue occurs when the model becomes overly dependent on the formatting or phrasing used in the training dataset.
For example, suppose all training examples follow the exact template:
### Instruction:... ### Response:...If every example looks identical, the model may learn to rely heavily on that pattern.
When users interact with the model in real-world scenarios, they may ask questions in completely different formats:
How does gradient descent work?
Explain gradient descent like I'm a beginner.
Could you describe gradient descent?
If the dataset lacks variation in instruction phrasing, the model may struggle to generalize.
One way to reduce this risk is to include instruction diversity during dataset creation.
Example variations:
Instruction:
Explain gradient descent.
Instruction:
Describe how gradient descent works in machine learning.
Instruction:
Give a beginner-friendly explanation of gradient descent.
Instruction:
What is gradient descent used for?
These variations teach the model that many different prompts can refer to the same task.
1.4.3 Catastrophic Forgetting
Large language models possess extensive knowledge from their pretraining phase. During fine-tuning, however, there is a risk that the model may lose some of this knowledge.
This phenomenon is known as catastrophic forgetting.
If the fine-tuning dataset is too narrow or too small, the model may adapt strongly to the new task distribution and lose general capabilities.
For example, imagine fine-tuning a model only on coding tasks.
After training, the model may become excellent at generating Python functions but noticeably worse at answering general knowledge questions or explaining scientific concepts.
To mitigate this risk, instruction datasets should maintain task diversity.
Mixing different task types helps preserve the broad abilities learned during pretraining.
Examples of mixed tasks include:
- Question answering
- Reasoning problems
- Coding tasks
- Text summarization
- Dialogue interactions
- Translation
Another strategy involves using low learning rates, which allow the model to adapt gradually without drastically changing its internal representations.
Example training configuration:
training_args = TrainingArguments( learning_rate=2e-5, num_train_epochs=3)Smaller learning rates reduce the likelihood of damaging previously learned knowledge.
1.4.4 Training Instability
Large models are sensitive to training hyperparameters. If these parameters are poorly chosen, training may become unstable.
Symptoms of instability include:
- Loss values that suddenly spike
- Training divergence
- Exploding gradients
- Extremely slow convergence
Several factors contribute to instability:
- Learning rates that are too high
- Batch sizes that are too small
- Poor dataset formatting
- Numerical precision issues
One widely used technique to stabilize training is gradient clipping.
Gradient clipping limits how large gradient values can become.
Example:
training_args = TrainingArguments( max_grad_norm=1.0)This prevents extreme gradient updates from destabilizing the model.
Another helpful strategy is learning rate warmup, which gradually increases the learning rate at the start of training.
1.4.5 GPU Memory Errors
Memory limitations are a practical challenge during fine-tuning.
Many developers encounter errors similar to:
CUDA out of memory.
These errors typically occur when:
- The batch size is too large
- The model is too large for available GPU memory
- Sequence lengths exceed expected limits
Several techniques can mitigate these issues:
- Reduce batch size
- Use gradient accumulation
- Enable mixed precision training
- Apply parameter-efficient fine-tuning (LoRA)
- Activate gradient checkpointing
Example configuration:
training_args = TrainingArguments( per_device_train_batch_size=1, gradient_accumulation_steps=16, fp16=True)Although smaller batches slow down training slightly, they allow models to fit within limited hardware constraints.
1.4.6 Data Leakage and Evaluation Bias
Another subtle issue arises when training and evaluation datasets overlap.
If the same examples appear in both sets, evaluation results may appear artificially strong.
For instance, if the model has already seen an instruction during training, it may simply memorize the response rather than demonstrate genuine understanding.
To prevent this, datasets should be carefully split into separate sets:
- Training set
- Validation set
- Test set
Example:
from sklearn.model_selection import train_test_split train_data, val_data = train_test_split(dataset, test_size=0.1)Maintaining clean dataset splits ensures that evaluation metrics reflect real model performance.
1.4.7 Misaligned Model Behavior
Even if training runs successfully, the resulting model may still behave in unexpected ways.
For example, a model might:
- Provide overly verbose answers
- Avoid answering certain questions
- Hallucinate information
- Ignore parts of the instruction
These issues often arise when the training dataset does not clearly demonstrate the desired behavior.
Instruction tuning teaches models by example. If the examples are inconsistent, the model will learn inconsistent behaviors.
Improving alignment often requires refining the dataset and possibly adding additional alignment techniques such as RLHF or DPO.
In practice, instruction tuning should be seen as an iterative process. Developers train a model, observe its behavior, adjust the dataset, and repeat the process.
Over time, this iterative refinement leads to models that follow instructions more reliably and produce responses that feel natural and helpful.
Understanding these potential pitfalls allows practitioners to design better datasets, more stable training pipelines, and ultimately more capable language models.