4.3 Measuring Hallucinations, Truthfulness, and Factual Grounding

In alignment work, hallucination is one of the most persistent failure modes. The challenge is that "hallucination" is not a single phenomenon, and you cannot measure it with a single metric. To evaluate it well, you need to separate what is true from what is supported.

Hallucinations emerge from the fundamental nature of language models: they are trained to predict plausible continuations, not to verify factual accuracy. During generation, the model samples from learned distributions over token sequences. When those distributions favor fluent-sounding but factually incorrect outputs—perhaps because similar patterns appeared frequently in training data, or because the model lacks knowledge in a particular domain—hallucinations occur. The model produces text that reads confidently and coherently while being partially or entirely false.

This creates a measurement problem: hallucinations vary in severity, detectability, and impact depending on the task and deployment context. A fabricated citation in a research assistant is qualitatively different from speculative reasoning in a creative writing tool, even though both involve generating unsupported content. Effective evaluation requires decomposing "hallucination" into specific, measurable failure modes.

4.3.1 What counts as a hallucination?

In practice, hallucinations tend to fall into three common categories:

• Fabricated facts: the model invents dates, names, numbers, or citations.
• Unsupported claims: the model introduces details that are not supported by the provided context (especially in retrieval-augmented systems).
• Overconfident speculation: the model guesses when it should express uncertainty or decline.

Each category requires different evaluation strategies because the failure mechanisms differ. Fabricated facts represent failures of parametric knowledge—the model either never learned the correct information or retrieves incorrect associations from its weights. Unsupported claims represent failures of grounding—the model ignores or misinterprets provided context in favor of its own generations. Overconfident speculation represents failures of calibration—the model fails to accurately estimate its own uncertainty.

These distinctions matter for alignment engineering because interventions that reduce one type of hallucination may not affect others. A model fine-tuned to better follow retrieval context might reduce unsupported claims while still fabricating facts when no context is provided. One trained to express uncertainty more frequently might reduce overconfident speculation while maintaining the same rate of factual fabrication when it does commit to an answer.

Measuring Hallucinations in Closed-Domain QA

For factual question answering, hallucination measurement is more straightforward because you can compare predictions against a reference answer set. Closed-domain QA provides ground truth: there are correct answers, and you can verify whether the model produces them.

Example:

def is_correct(prediction, reference):    return prediction.strip().lower() == reference.strip().lower()

COPY

But hallucination detection must go beyond exact match. Exact string comparison fails to capture semantic equivalence and penalizes correct answers that include additional true information or use different phrasing.

For example:

Question: "Who wrote 1984?"

Model answer: "George Orwell wrote 1984 in 1948."

The extra detail is correct — but exact match would fail. The model has provided a factually accurate, more informative response than simply "George Orwell," yet a naive evaluation metric would mark it as incorrect due to the additional tokens.

This illustrates a broader evaluation challenge: the relationship between completeness and correctness is not straightforward. Additional details can be helpful elaborations, irrelevant digressions, or subtle hallucinations. A response that says "George Orwell wrote 1984 in 1949" would also fail exact match, but for a different reason—it contains a factual error that could mislead users.

A better approach combines:

• Token overlap metrics
• Semantic similarity models
• Human verification

Token overlap metrics like F1 score provide a middle ground between exact match and pure semantic similarity. They reward partial matches and are robust to minor phrasing variations, though they still struggle with paraphrase and can be gamed by models that learn to echo parts of the question.

Semantic similarity models offer a more flexible approach by measuring meaning rather than surface form. They can recognize that "George Orwell" and "Eric Arthur Blair" refer to the same person, and that "authored" and "wrote" are equivalent in this context.

Example using semantic similarity (conceptual):

from sentence_transformers import SentenceTransformer, util model = SentenceTransformer("all-MiniLM-L6-v2") def semantic_similarity(a, b):    emb1 = model.encode(a, convert_to_tensor=True)    emb2 = model.encode(b, convert_to_tensor=True)    return util.cos_sim(emb1, emb2).item()

COPY

High similarity suggests factual alignment. However, semantic similarity has its own limitations: it can be fooled by responses that are topically related but factually incorrect, and it provides a continuous score rather than a binary judgment, requiring you to set thresholds that may vary across question types.

Human verification remains the gold standard for nuanced cases. Humans can judge whether additional details are correct, whether paraphrases preserve meaning, and whether responses that don't exactly match the reference are nonetheless acceptable. But human evaluation is expensive and doesn't scale to continuous monitoring of production systems.

In practice, comprehensive QA hallucination evaluation combines all three approaches: automated metrics for rapid iteration and regression testing, semantic similarity for capturing meaning beyond surface form, and sampled human review for validating that automated metrics align with actual quality. The specific balance depends on your evaluation budget and the consequences of different error types in your deployment context.

4.3.2 Truthfulness vs grounding

These terms are closely related but capture fundamentally different dimensions of model reliability. Understanding the distinction is essential for designing evaluation strategies that match your deployment needs.

• Truthfulness asks: Is the content consistent with real-world facts?
• Example: "What is the capital of Australia?" → Canberra.
• Truthfulness evaluation requires external verification against ground truth knowledge bases, reference datasets, or expert judgment. The model's internal reasoning or confidence is irrelevant—only factual correctness matters.
• Grounding asks: Is the content supported by the evidence the model was given in this interaction?
• Example: In a RAG pipeline, are the claims supported by retrieved passages?
• Grounding evaluation focuses on attribution and evidence alignment. A grounded response must be derivable from the provided context, regardless of whether that context is itself factually correct. This makes grounding verification a tractable computational problem: you can check entailment between generated text and source documents without needing access to external truth.

This distinction creates three possible failure modes, each with different implications for system reliability:

• Truthful but not grounded: The model generates factually correct information that is not present in the provided context. This occurs when the model draws on its parametric knowledge rather than adhering strictly to retrieval context. Whether this constitutes a failure depends on your application. In some systems, you want the model to augment retrieved context with its own knowledge when context is incomplete. In others—particularly in high-stakes domains like legal or medical applications—you need strict attribution to prevent the model from introducing unverifiable claims, even if those claims happen to be true.
• Grounded but incomplete or misleading: The model only states what appears in the context, but omits critical information or presents it in a way that misrepresents the source material. For example, if retrieved context mentions both benefits and risks of a treatment, a response that only cites the benefits is grounded in a narrow technical sense but fails to faithfully represent the evidence. This highlights why grounding alone is insufficient—you also need to evaluate comprehensiveness and whether the model's synthesis introduces bias through selective citation.
• Neither truthful nor grounded: The model fabricates information that contradicts both the provided context and external facts. This represents the most severe failure mode and often indicates fundamental problems with instruction-following or context adherence during fine-tuning.

In many production RAG systems, grounding is the first-order requirement because it is auditable: you can trace claims back to evidence. This traceability serves multiple purposes. It allows users to verify the model's reasoning, provides legal and compliance teams with documentation of how conclusions were reached, and creates opportunities for automated validation that scale beyond what human review can achieve.

However, prioritizing grounding over truthfulness comes with trade-offs. A model trained to strictly adhere to retrieved context may refuse to answer questions when context is incomplete, even if it possesses relevant parametric knowledge. It may also propagate errors present in the retrieval corpus rather than correcting them using its broader knowledge. The appropriate balance depends on your risk tolerance: systems where unverifiable claims create legal liability should favor strict grounding, while systems where helpfulness and coverage matter more may benefit from allowing the model to supplement context with parametric knowledge when appropriate.

Measuring grounding typically involves:

• Entailment checking between generated claims and source documents
• Citation validation to ensure referenced passages actually support attributed claims
• Claim-level decomposition to verify that each factual assertion can be traced to evidence

A practical grounding verification approach might look like this:

def verify_grounding(claim, context_passages, entailment_model):    """    Verify if a claim is supported by provided context.    Returns: (is_grounded, supporting_passage_id, confidence)    """    for idx, passage in enumerate(context_passages):        # Check if passage entails the claim        result = entailment_model.predict(            premise=passage,            hypothesis=claim        )                if result['label'] == 'entailment' and result['confidence'] > 0.8:            return True, idx, result['confidence']        return False, None, 0.0 # Example usage with claim decompositiondef evaluate_response_grounding(response, context_passages):    """Decompose response into claims and verify each."""    claims = extract_factual_claims(response)  # Use claim extraction model        grounding_scores = []    for claim in claims:        is_grounded, passage_id, conf = verify_grounding(            claim, context_passages, entailment_model        )        grounding_scores.append({            'claim': claim,            'grounded': is_grounded,            'source': passage_id,            'confidence': conf        })        # Overall grounding rate    grounding_rate = sum(s['grounded'] for s in grounding_scores) / len(claims)    return grounding_rate, grounding_scores

COPY

Let's break down this grounding verification implementation:

Core Function: verify_grounding

• Purpose: Checks whether a single claim is supported by any passage in the provided context.
• Parameters:
• claim: A single factual assertion extracted from the model's response
• context_passages: List of retrieved documents or text snippets that should support the claim
• entailment_model: A natural language inference (NLI) model that determines whether a premise logically entails a hypothesis
• Logic:
• Iterates through each context passage
• Treats the passage as the premise and the claim as the hypothesis
• Uses the entailment model to predict whether the passage supports the claim
• Returns True if any passage entails the claim with confidence above 0.8
• Returns False if no supporting passage is found
• Return values:
• is_grounded: Boolean indicating whether the claim is supported
• supporting_passage_id: Index of the passage that supports the claim (or None)
• confidence: The entailment model's confidence score

Wrapper Function: evaluateresponsegrounding

• Purpose: Evaluates the grounding of an entire response by decomposing it into individual claims.
• Process:
• extract_factual_claims(response): Uses a claim extraction model to break the response into atomic factual statements. This is critical because responses often contain multiple claims that may have different grounding statuses.
• For each extracted claim, calls verify_grounding to check support
• Collects detailed results for each claim (whether grounded, which source supports it, confidence level)
• Computes an overall grounding rate: the fraction of claims that are supported by context
• Output:
• grounding_rate: A scalar metric (0.0 to 1.0) representing overall response quality
• grounding_scores: Detailed breakdown enabling inspection of which specific claims failed grounding checks

Key Design Decisions

• Confidence threshold (0.8): This is a tunable parameter. Higher thresholds reduce false positives (incorrectly marking unsupported claims as grounded) but increase false negatives. The appropriate threshold depends on your risk tolerance and the quality of your entailment model.
• Claim decomposition: This is essential because a response like "Paris is the capital of France and was founded in the 3rd century BC" contains two claims with potentially different grounding statuses. Without decomposition, you cannot identify which specific assertions are problematic.
• Short-circuit evaluation: The function returns as soon as it finds supporting evidence, rather than checking all passages. This improves efficiency when context sets are large.
• Traceability: By returning the supporting passage ID, the system enables auditing and allows users to verify the model's reasoning by examining the cited evidence.

Limitations and Extensions

• Entailment model quality: This approach is only as good as the underlying NLI model. Modern entailment models can struggle with numerical reasoning, temporal logic, and negation.
• Claim extraction accuracy: If the claim extraction model misses important assertions or incorrectly segments claims, grounding evaluation will be incomplete.
• Context ordering: The function does not consider that some passages may be more authoritative than others. In production systems, you might want to weight passages by retrieval score or source credibility.
• Contradiction detection: This implementation only checks for entailment. It does not detect when a claim contradicts the provided context, which is a distinct failure mode that may require separate handling.

This approach decomposes the response into individual factual claims, then verifies each against the provided context using an entailment model. The grounding rate captures what fraction of claims are supported by evidence, while the detailed scores enable debugging specific grounding failures.

The critical insight is that truthfulness and grounding measure different things, and optimizing for one does not guarantee the other. A comprehensive evaluation framework must measure both dimensions and explicitly decide which takes priority in your deployment context.

Most of the mechanics for measuring QA hallucination, grounding violations, and calibration (exact match vs semantic grading, sentence/claim checks, entailment-style verification, uncertainty scoring scaffolds) are covered in Section 4.2.1. This section focuses on the conceptual distinctions and the most common evaluation setups.

4.3.3 Truthfulness-focused benchmarks and protocols

Several widely used benchmarks and evaluation protocols have been developed specifically to measure truthfulness and factual reliability in language models. These resources represent different approaches to the fundamental challenge of detecting when models generate plausible-sounding but incorrect information:

• TruthfulQA: This benchmark probes whether models repeat common misconceptions by asking questions where the truthful answer contradicts widely held false beliefs. For example, it might ask "What happens if you crack your knuckles a lot?" where the truthful answer contradicts the popular misconception that it causes arthritis. The benchmark is designed to expose the gap between what a model has learned from text corpora (which contain many misconceptions) and what is actually true. Models that perform well on general knowledge tasks often struggle on TruthfulQA because they have learned to reproduce common patterns in text rather than evaluate factual accuracy.
• FactScore: This metric evaluates factual accuracy by decomposing generated text into atomic claims and verifying each claim against authoritative sources. Rather than treating a response as correct or incorrect as a whole, FactScore computes the percentage of claims that can be verified. This granular approach is particularly valuable for long-form generation where a response might contain many factual statements with varying accuracy. The metric's strength is its ability to identify where in a response factual errors occur, enabling targeted improvement during fine-tuning.
• SelfCheckGPT and consistency-based methods: These approaches estimate factual risk without requiring external knowledge bases by exploiting a key insight: if a model truly knows a fact, it will generate consistent answers across multiple samples. The method works by sampling multiple responses to the same prompt and measuring agreement. High variance across samples suggests the model is confabulating rather than retrieving reliable knowledge. This approach is particularly practical because it requires no human annotation or external knowledge sources—you can estimate hallucination risk using the model itself.

Beyond these specific tools, a practical lesson from truthfulness research is that effective evaluation benefits from adversarial design. Standard evaluation sets often inadvertently favor models that have memorized common knowledge, but fail to test whether models can distinguish truth from plausible-sounding falsehoods. Adversarial evaluation deliberately includes:

• Popular myths: Questions where the most common answer in training data is incorrect (e.g., "Do we only use 10% of our brains?")
• Leading questions: Prompts that presuppose false information and see whether the model pushes back (e.g., "What health benefits come from the toxins released during a detox cleanse?")
• Ambiguous phrasing: Questions that could be interpreted multiple ways, testing whether the model recognizes uncertainty or confidently answers based on one interpretation
• "Trap" prompts that invite plausible-sounding fabrication: Requests for specific facts that do not exist, such as "What did Einstein say about quantum computing?" (a technology that emerged after his death). These prompts test whether models will invent plausible-sounding but false information when they lack knowledge.

The value of adversarial design extends beyond benchmark creation. When constructing evaluation sets for your specific domain, deliberately including cases where correct behavior requires resisting plausible-sounding errors will reveal model weaknesses that standard evaluation misses. This is particularly important after fine-tuning, where models may become overconfident in domains where their training data contains systematic biases or gaps.

4.3.4 Detecting fabricated citations (a common, high-impact case)

A frequent hallucination pattern is invented references (papers that do not exist, broken DOIs, or URLs that do not resolve). Even a simple automated check helps catch obvious failures.

This failure mode is particularly insidious because citations carry epistemic weight. When a model provides a reference, users reasonably interpret this as evidence that the claim is grounded in verifiable sources. Fabricated citations exploit this trust, creating an illusion of rigor while actually increasing the risk of misinformation propagation. Unlike other hallucination types that may be caught through surface-level inconsistencies, invented references often have plausible formatting—they look like real academic citations, complete with author names, publication years, and journal titles. The model has learned the structure of citations from training data without acquiring the ability to verify their existence.

The practical impact varies by domain. In academic research contexts, fabricated citations can derail literature reviews and waste researcher time chasing nonexistent sources. In medical or legal applications, invented references to authoritative sources create liability risks. Even in lower-stakes contexts, citation fabrication erodes user trust once discovered.

A minimal pipeline often includes:

• Validating that cited URLs resolve.
• Validating that DOIs resolve.
• Sampling a subset for human verification (because "resolves" does not mean "supports the claim").

URL Resolution Validation

The first line of defense is verifying that URLs and DOIs point to actual resources. This catches the most egregious cases where models generate syntactically valid but entirely fictional identifiers.

Simple URL check example:

import requests def check_url_exists(url):    try:        response = requests.head(url, timeout=5)        return response.status_code < 400    except:        return False

COPY

This basic implementation uses HTTP HEAD requests (which retrieve only headers, not full content) to verify accessibility. A status code below 400 indicates success. The timeout prevents hanging on unresponsive URLs.

For production systems, you should extend this foundation with:

• DOI resolution through official APIs: Services like doi.org and CrossRef provide APIs that return metadata for valid DOIs. This is more reliable than simple URL checking because DOIs are persistent identifiers maintained by registration agencies.
• Retry logic with exponential backoff: Temporary network issues or rate limiting can cause false negatives. Implementing retry mechanisms with increasing delays reduces spurious validation failures.
• Cache validation results: If you're evaluating multiple model outputs that may reference the same sources, caching validation results prevents redundant network requests and improves efficiency.
• Handle redirects appropriately: Many academic publishers use redirects. Your validation logic should follow redirects and verify the final destination, not just the initial URL.
• Distinguish error types: Not all validation failures are equal. A 404 (not found) strongly suggests fabrication. A 403 (forbidden) or 429 (rate limited) may indicate access restrictions rather than nonexistence. Your evaluation pipeline should categorize these differently.

Beyond Resolution: Content Verification

URL resolution is necessary but insufficient. A URL that resolves does not guarantee that the cited source supports the claim. The model might cite a real paper about a completely unrelated topic, or misrepresent the paper's findings.

For large-scale evaluation, automated citation validation is essential.However, a complete validation strategy requires sampling for human verification. A practical approach:

• Automatic filtering: Use URL/DOI resolution to eliminate obvious fabrications and reduce the review set.
• Stratified sampling: Manually verify a representative subset of resolved citations, stratifying by domain, source type, or other relevant factors to ensure coverage of different citation patterns.
• Content alignment scoring: For high-priority applications, implement automated relevance checks by retrieving citation content and using semantic similarity metrics to estimate whether the cited source likely supports the claim. This won't replace human judgment but can prioritize which citations most urgently need manual review.
• Track citation patterns: Monitor which types of sources the model tends to cite correctly versus incorrectly. If the model reliably hallucinates citations to specific journals or from particular time periods, this suggests systematic issues in training data or knowledge gaps that fine-tuning might address.

Integration with Model Development

Citation validation should not be viewed purely as a post-hoc evaluation step. The insights it provides should feed back into model development:

• If a model frequently fabricates citations in a specific domain, this indicates a knowledge gap where retrieval-augmented generation or targeted fine-tuning may help.
• Tracking fabrication rates across model versions reveals whether alignment or fine-tuning interventions improve factual grounding or inadvertently increase overconfident citation behavior.
• Citation validation data can be used to construct training examples for teaching models to say "I don't have a specific source for this claim" rather than inventing references.

The broader principle is that citation fabrication represents a measurable, high-impact hallucination type where simple automation provides substantial value. While comprehensive truthfulness evaluation requires sophisticated approaches, even basic validation infrastructure catches failure modes that would otherwise undermine model credibility in domains where source attribution matters.

4.3.5 Trade-off: helpfulness vs hallucination vs refusal

Reducing hallucination often pushes the model toward:

• More hedging ("this might be the case" rather than "this is the case")
• More "I don't know" responses when uncertain
• More refusals in borderline cases where the model lacks confidence

This represents a fundamental tension in language model behavior. When you optimize a model to avoid making factual errors, you implicitly teach it to be more conservative. The model learns that confident assertions carry risk, so it hedges more frequently. It learns that providing an answer when uncertain leads to negative feedback, so it refuses more often. The result is a model that makes fewer mistakes—but also provides fewer direct answers.

There is no universal optimum. The right trade-off depends entirely on deployment context and the relative costs of different error types. Consider three distinct scenarios:

• A medical assistant should favor caution. In healthcare applications, the cost of confidently stating incorrect medical information far exceeds the cost of refusing to answer or hedging. A patient who receives fabricated medical advice may make dangerous health decisions. A patient who receives a hedged response or a refusal will likely seek information from other sources. The asymmetry is clear: false confidence causes harm, while appropriate caution merely reduces convenience.
• A creative writing tool can tolerate more speculation. When helping users brainstorm story ideas or generate creative content, hallucination is less problematic—in fact, unexpected or novel suggestions may even be valuable. If the model suggests a historical detail for a fiction story and that detail is inaccurate, the user can verify or ignore it without significant consequence. Excessive hedging or frequent refusals would disrupt the creative flow and reduce the tool's utility. In this context, the cost of hallucination is low while the cost of over-cautious behavior is high.
• A research assistant should prioritize evidence-backed claims and citations. For academic or professional research support, the model should only make claims it can support with verifiable sources. However, complete refusal to engage with complex questions would render the tool useless. The optimal behavior might involve providing well-sourced answers when possible, explicitly noting uncertainty when evidence is mixed, and refusing only when the question falls entirely outside the model's knowledge base.

The challenge becomes measurable when you instrument your evaluation to capture these trade-offs explicitly. Rather than treating "hallucination rate" and "refusal rate" as independent metrics to minimize, you should measure them jointly and understand their relationship. A practical approach involves tracking:

• Answer rate: What percentage of questions receive substantive answers versus refusals or non-responses?
• Conditional accuracy: Among questions that receive substantive answers, what percentage are factually correct?
• Refusal appropriateness: Are refusals concentrated on questions where the model genuinely lacks knowledge, or is the model refusing answerable questions unnecessarily?

This three-dimensional view reveals whether your model is finding the right trade-off. A model with 95% conditional accuracy but only 30% answer rate may be too conservative. A model with 90% answer rate but 70% conditional accuracy may be too aggressive. The appropriate balance depends on your deployment context.

During alignment, you can directly tune this trade-off through your preference data. If you want to reduce over-refusal, include examples in your training set where:

• The chosen response provides a well-hedged but informative answer to a difficult question
• The rejected response refuses to engage with the question despite having relevant knowledge

Conversely, if you want to increase caution, include examples where:

• The chosen response acknowledges uncertainty or refuses to speculate
• The rejected response provides a confident but unsupported claim

The relative proportion of these example types in your training data directly influences where the model lands on the helpfulness-accuracy-refusal spectrum.

Reflection question: If a DPO-aligned chatbot reduces hallucinations from 15% to 8% but increases refusals from 5% to 18%, is that a net improvement? The answer is not purely technical. It is a deployment decision that requires understanding your users' needs and the consequences of different error types in your application context. A medical advisor and a creative writing assistant would answer this question differently—and should be aligned accordingly.