Step 7: Implement Hallucination Detection (Small but Useful)

This is crucial. If you do not constrain the model, it may use outside knowledge and your "hallucination detector" becomes meaningless.

Why does this matter so much? Because language models are trained on vast corpora of text, and they've internalized enormous amounts of world knowledge. When you ask "How long does the warranty last?" without strict grounding constraints, the model doesn't just look at your context—it also draws on everything it's seen about warranties during pretraining. It might "know" that consumer electronics typically have one-year warranties, that automotive warranties are often longer, that extended warranties are a common upsell. All of this prior knowledge leaks into the response unless you explicitly block it.

This knowledge contamination makes evaluation useless. If your base model hallucinates 30% of the time and your fine-tuned model hallucinates 15% of the time, is that because fine-tuning improved grounding, or because fine-tuning happened to align the model's priors more closely with the specific domain of your test set? You can't tell. The model might be relying on context more often, or it might just be making luckier guesses. Without strict prompting that forces context-only reasoning, you're measuring a blend of grounding ability and domain knowledge overlap—and only the former is what you actually care about.

The solution is to make grounding constraints explicit and unambiguous in every single prompt. Don't trust the model to "figure out" that it should stick to context. Don't assume that because you provided context, the model will prioritize it over memorized knowledge. Models don't have a built-in notion of "use this and only this"—they're pattern matchers that blend all available signals unless you tell them otherwise.

A strong prompt format includes three essential components:

• Provide context clearly and explicitly, separated from the question
• Instruct: "Use only the context. If not present, say you don't know."
• Require a short answer style to discourage elaboration beyond what's supported

The separation between context and question is structural scaffolding that makes it easier for the model to distinguish "here's what I should rely on" from "here's what I'm being asked." By placing context in a clearly marked section, you're creating a boundary. This is especially important for smaller models, which are more prone to conflating instruction-following (the meta-task of "answer the question") with content-following (the object-task of "use only this information").

The explicit instruction to say "I don't know" is your escape hatch. Without it, models default to their pretraining behavior: always produce a plausible-sounding answer, even when uncertain. This is because they were trained on a corpus where nearly every question has an answer somewhere in the training data. The model has no inherent concept of epistemic humility—it must be taught that "I don't know" is a valid and often correct response. By including this instruction, you're giving the model permission to acknowledge the limits of the provided context rather than filling gaps with invention.

The requirement for short answers serves as a hedge against elaboration drift. Models love to elaborate. Given a simple factual question, they'll often provide the answer and then add context, caveats, related information, or helpful asides. Most of this additional content comes from world knowledge, not from your provided context. By requesting brevity, you're cutting off the model's tendency to keep generating once it's answered the core question. This isn't foolproof—models can still hallucinate in short responses—but it reduces the surface area for unsupported claims.

Example template:

def grounded_prompt(context, question):    return f"""You are a helpful assistant.Use ONLY the context below to answer the question.If the answer is not in the context, say "I don't know based on the provided context." Context:{context} Question:{question} Answer:"""

COPY

This template is deliberately repetitive. The instruction appears twice in slightly different forms: once in the imperative ("Use ONLY the context") and once in the conditional ("If the answer is not in the context"). This redundancy is intentional. Instruction-following in language models is probabilistic, not deterministic. A single instruction might be ignored or misinterpreted, especially under distribution shift (when your test questions don't quite match the model's fine-tuning data). By repeating the constraint in different phrasings, you increase the likelihood that at least one formulation resonates with the model's learned behavior patterns.

The all-caps "ONLY" is another intentional choice. While models don't technically parse capitalization as emphasis the way humans do, capitalization does change the token distribution in ways that can affect attention patterns. In the model's training data, capitalized words often appear in contexts where emphasis matters—warnings, legal disclaimers, critical instructions. By capitalizing "ONLY," you're shifting the prompt slightly toward that distribution, which may marginally increase the model's tendency to treat this as a hard constraint rather than a suggestion.

The specific phrasing "I don't know based on the provided context" is more precise than just "I don't know." The latter could mean many things: the model is uncertain, the question is ambiguous, the model refuses to answer for safety reasons. The former is unambiguous: the information needed to answer is not present in the given text. This precision matters for evaluation. When you count "I don't know" responses, you want to count appropriate refusals (cases where the context genuinely doesn't support an answer), not confused non-answers or overly cautious safety refusals.

One additional consideration: this prompt format works best when the model has been fine-tuned or few-shot trained with similar formatting. If your model has never seen this structure during training, it may not follow the instructions reliably. Ideally, your fine-tuning data includes many examples of context-grounded question answering with explicit grounding constraints. If you're evaluating a base model that hasn't seen this format, expect higher failure rates—not necessarily because the model can't ground, but because it hasn't learned to interpret these specific instructions as binding constraints.