Evaluating RAG systems: faithfulness, relevance, and hallucination rate

A retrieval-augmented generation (RAG) system can produce an answer that sounds authoritative, responds directly to the user’s question, and includes citations—while being completely wrong.

This is one of the most consequential failure modes in production RAG applications.

A language model might retrieve an irrelevant document, misinterpret a relevant passage, invent a detail, or confidently repeat inaccurate information found in its sources. Worse, the resulting answer might pass some automated evaluations despite being factually incorrect.

Traditional language model benchmarks do not fully capture these failures. Even a high semantic similarity score between a generated answer and a reference answer tells us little about whether the response is supported by the documents the model retrieved.

To evaluate RAG reliably, we need to measure several independent properties:

• Faithfulness: Are the answer’s factual claims supported by retrieved evidence?

• Answer relevance: Does the response address the user’s question?

• Hallucination rate: What proportion of generated claims cannot be supported by retrieved evidence?

• Answer correctness: Does the response agree with independently verified facts?

• Retrieval quality: Did the system retrieve the necessary documents, and how much irrelevant information did it include?

These measurements reveal fundamentally different problems.

A highly relevant answer can be unfaithful. A faithful answer can be factually incorrect. A retriever can find the right evidence while the generator ignores it.

In this tutorial, we will build a reproducible RAG evaluation pipeline using synthetic e-commerce policies, calculate claim-level metrics, conduct a controlled experiment, and connect our implementation to the current Ragas evaluation framework.

We will also correct several common implementation mistakes, including inconsistent retrieval cutoffs, deprecated library imports, incompatible model parameters, and quality gates that confuse deliberately failing test fixtures with deployment candidates.

By the end, you will have a practical foundation for measuring RAG reliability, investigating failures, and preventing regressions.

1. Why RAG evaluation requires multiple dimensions

A typical RAG application contains three main stages:

1. Retrieval: Find information relevant to a question.

2. Generation: Produce an answer using the retrieved information.

3. Presentation: Return the answer, often with citations or source references.

Each stage introduces different failure modes.

Consider a customer-support assistant answering:

How long do customers have to return unopened products?

The authoritative return policy specifies 30 days.

Now consider three scenarios.

Scenario A: Retrieval succeeds, generation fails

The retriever finds the correct policy document:

Unopened products may be returned within 30 days.

The language model responds:

Customers have 60 days to return unopened products.

Retrieval worked, but generation failed.

The answer contradicts the retrieved evidence. Faithfulness should therefore be low, even though retrieval recall is high.

Scenario B: Retrieval fails, generation guesses correctly

The retriever returns an unrelated shipping document.

The model nevertheless responds:

Customers have 30 days to return unopened products.

The answer is factually correct, perhaps because the model learned something similar during training.

However, the retrieved evidence does not support that answer.

This demonstrates why correctness and faithfulness must remain separate.

A correct guess is not equivalent to an answer grounded in auditable evidence.

Scenario C: Retrieval finds misinformation

The retriever finds an outdated policy mirror claiming that customers have 90 days to return products.

The model responds:

Customers have 90 days to return unopened products.

The answer accurately reflects the retrieved document but contradicts the authoritative policy.

This produces an important result:

Faithfulness can be perfect while factual correctness is zero.

Improving generation alone may not fix this problem. The information entering the generation stage is already wrong.

The rest of this tutorial makes these distinctions measurable.

2. Define the evaluation metrics precisely

Before writing code, we need definitions that can be translated into calculations.

Some metrics can be computed deterministically from labeled examples. Others require language models, embedding models, or human judgments.

We should never confuse these approaches.

2.1 Faithfulness: Is the answer grounded in evidence?

Faithfulness measures whether the factual claims in a response can be inferred from retrieved context.

A common claim-based definition is:

Faithfulness = supported claims / total generated claims

Suppose a shipping assistant responds:

Express shipping takes two business days, costs nothing, and includes insurance.

The retrieved document says:

Express shipping takes two business days.

We can decompose the response into three atomic claims.

Faithfulness = 1 / 3 = 0.333

The response is approximately 33.3% faithful.

The important detail is claim decomposition.

Evaluating the entire response as one statement could hide two unsupported assertions inside an otherwise accurate sentence.

A stronger evaluator first extracts atomic propositions, then checks whether each is supported by the available evidence.

The Ragas faithfulness documentation describes this general claim-generation and verification approach.

2.2 Answer relevance: Does the response answer the question?

Answer relevance measures whether the generated response satisfies the user’s information need.

Consider:

How quickly does express delivery arrive?

Response A:

Express delivery takes two business days.

Response B:

We offer express shipping, premium packaging, gift messages, and recyclable materials.

Response B discusses related subjects but does not answer the question.

Keyword overlap is not enough.

The Ragas answer-relevancy approach generates hypothetical questions from the response and compares their embeddings with the original user question.

This estimates semantic alignment without verifying factual correctness.

A strong relevance evaluator should consider:

• Whether the requested information is present.

• Whether every requested subquestion is addressed.

• Whether the response includes distracting content.

• Whether the response appropriately abstains when information is unavailable.

• Whether conversational context changes the meaning of the question.

A response can be relevant and completely incorrect.

2.3 Hallucination rate: How many claims lack evidence?

For our experiment, we define evidence-grounding hallucination rate as:

Hallucination rate = unsupported claims / total generated claims

Under binary claim labeling:

Hallucination rate = 1 − faithfulness

This identity holds only when both metrics use the same extracted claims, evidence, and support judgments.

It is not a universal relationship across evaluation frameworks.

More advanced evaluators should distinguish three possibilities:

A claim absent from the retrieved documents is not necessarily false.

Conversely, a false claim copied faithfully from an inaccurate document may receive a perfect grounding score.

For production reporting, separate:

• Evidence-grounding failure rate: Claims unsupported by retrieval.

• Factual error rate: Claims contradicting independently verified facts.

These measure different risks.

2.4 Answer correctness: Is the answer true?

Answer correctness compares generated claims against an authoritative reference.

It does not rely exclusively on the documents retrieved for the request.

For structured facts, we can calculate:

Precision = correct generated claims / generated claims

Recall = correct generated claims / reference claims

F1 = 2 × precision × recall / (precision + recall)

F1 rewards both factual accuracy and completeness.

Suppose the reference answer contains two facts:

• Delivery takes two business days.

• Tracking is included.

The generated answer contains:

• Delivery takes two business days.

• Shipping is free.

One of the two generated claims matches the reference, and one of the two reference claims is recovered.

Precision = 0.5

Recall = 0.5

F1 = 0.5

Real natural-language evaluation requires semantic reasoning because equivalent facts can be expressed using different wording.

Ragas provides a factual-correctness metric that decomposes and compares factual statements.

2.5 Retrieval precision and recall at a fixed k

Retrieval metrics help distinguish search failures from generation failures.

For a fixed retrieval cutoff k:

Precision@k = relevant documents among the top k / k

Recall@k = relevant documents among the top k / total known relevant documents

Suppose four documents are relevant to a question.

The retriever returns five results, including three relevant documents.

Precision@5 = 3 / 5 = 0.60

Recall@5 = 3 / 4 = 0.75

Precision describes retrieval noise.

Recall measures how much relevant evidence was recovered.

However, one implementation detail matters enormously:

The cutoff k must remain consistent across comparable evaluation examples.

It is incorrect to label a metric Precision@5 while dividing by the number of documents returned in each individual trace.

That produces a variable-cutoff metric disguised as a fixed-cutoff measurement.

For our experiment, we will use k = 2.

Every benchmark case will contain two ranked retrieved documents. Our evaluator will score only those first two positions.

In a production system that occasionally returns fewer than k documents, you must explicitly define whether empty positions count against precision or whether you report a separate variable-denominator measure.

Ranking-sensitive metrics such as average precision and normalized discounted cumulative gain provide additional information beyond basic precision and recall.

3. Build a reproducible evaluation dataset

We will construct a fictional e-commerce knowledge base.

The data covers:

• Product returns.

• Express delivery.

• Customer support.

• Packaging.

Everything is synthetic.

The initial experiment requires no external dataset, API key, vector database, or language model.

Our test cases deliberately include five different behaviors.

These cases form a small diagnostic test fixture rather than a representative production benchmark.

3.1 Prepare the environment

Use Python 3.11 or newer.

Create a project directory:

mkdir rag-evaluation-lab
cd rag-evaluation-lab
python -m venv .venv

Activate the virtual environment on macOS or Linux:

source .venv/bin/activate

The deterministic implementation uses only Python’s standard library.

Our completed project will contain the following files:

rag-evaluation-lab/
├── .venv/
├── rag_eval.py
├── regression_tests.py
├── candidate_gate.py
├── make_demo_candidate.py
├── candidate_traces.json
└── ragas_live.py

The final file is an optional hosted-model integration.

3.2 Define the documents and test cases

Create rag_eval.py.

Begin with the knowledge base:

from dataclasses import dataclass
from statistics import mean

K = 2
Fact = tuple[str, str]

DOCUMENTS = {
    "returns_current": {
        "text": (
            "Official policy: unopened products may be "
            "returned within 30 days."
        ),
        "facts": {
            ("return_window", "30 days"),
        },
    },
    "returns_mirror": {
        "text": (
            "Unofficial mirror: unopened products may be "
            "returned within 90 days."
        ),
        "facts": {
            ("return_window", "90 days"),
        },
    },
    "shipping": {
        "text": (
            "Express delivery takes 2 business days "
            "and includes tracking."
        ),
        "facts": {
            ("express_eta", "2 business days"),
            ("tracking", "included"),
        },
    },
    "support": {
        "text": (
            "Live chat operates weekdays "
            "from 09:00 to 17:00."
        ),
        "facts": {
            ("chat_hours", "weekdays 09:00-17:00"),
        },
    },
    "packaging": {
        "text": "Gift boxes are recyclable.",
        "facts": {
            ("gift_boxes", "recyclable"),
        },
    },
}

Each document contains natural-language text and structured facts.

A fact is represented as an attribute-value pair.

For example, the attribute return_window can have the value 30 days or 90 days.

Structured facts allow deterministic scoring without ambiguity about paraphrases.

The incorrect mirror is intentionally included to demonstrate the limits of grounding metrics.

Next, define our evaluation trace:

@dataclass(frozen=True)
class Trace:
    name: str
    question: str
    response: str
    retrieved: tuple[str, ...]
    relevant_documents: tuple[str, ...]
    expected_claims: tuple[Fact, ...]
    response_claims: tuple[Fact, ...]
    requested_facets: tuple[str, ...]

Each field has a specific purpose.

• question: The user’s original request.

• response: The generated answer.

• retrieved: Ranked document identifiers.

• relevant_documents: Human-labeled relevant documents.

• expected_claims: Independently verified correct facts.

• response_claims: Atomic claims extracted from the response.

• requested_facets: Information the question asks for.

The distinction between expected claims and response claims is essential.

Expected claims represent ground truth. Response claims represent what the model actually asserted.

Now add the benchmark cases:

TRACES = [
    Trace(
        name="good_return",
        question="How long can I return an unopened product?",
        response="You can return unopened products within 30 days.",
        retrieved=("returns_current", "packaging"),
        relevant_documents=("returns_current",),
        expected_claims=(("return_window", "30 days"),),
        response_claims=(("return_window", "30 days"),),
        requested_facets=("return_window",),
    ),
    Trace(
        name="extra_claim",
        question="How quickly does express delivery arrive?",
        response="Express delivery takes 2 business days and is free.",
        retrieved=("shipping", "packaging"),
        relevant_documents=("shipping",),
        expected_claims=(("express_eta", "2 business days"),),
        response_claims=(
            ("express_eta", "2 business days"),
            ("shipping_price", "free"),
        ),
        requested_facets=("express_eta",),
    ),
    Trace(
        name="lucky_guess",
        question="When is live chat available?",
        response="Live chat operates weekdays from 09:00 to 17:00.",
        retrieved=("packaging", "returns_mirror"),
        relevant_documents=("support",),
        expected_claims=(
            ("chat_hours", "weekdays 09:00-17:00"),
        ),
        response_claims=(
            ("chat_hours", "weekdays 09:00-17:00"),
        ),
        requested_facets=("chat_hours",),
    ),
    Trace(
        name="wrong_mirror",
        question="How long can I return an unopened product?",
        response="You can return unopened products within 90 days.",
        retrieved=("returns_mirror", "packaging"),
        relevant_documents=("returns_current",),
        expected_claims=(("return_window", "30 days"),),
        response_claims=(("return_window", "90 days"),),
        requested_facets=("return_window",),
    ),
    Trace(
        name="off_topic",
        question="Does express delivery include tracking?",
        response="Gift boxes are recyclable.",
        retrieved=("packaging", "shipping"),
        relevant_documents=("shipping",),
        expected_claims=(("tracking", "included"),),
        response_claims=(("gift_boxes", "recyclable"),),
        requested_facets=("tracking",),
    ),
]

Notice two important design decisions.

First, every trace contains exactly two retrieved documents, maintaining our fixed k = 2 cutoff.

Second, retrieval order is preserved.

The off_topic case retrieves packaging first and shipping second. Both documents are available to the generator, but the response uses the irrelevant packaging fact.

This allows us to distinguish evidence availability from appropriate evidence usage.

4. Implement deterministic RAG evaluation metrics

Our first evaluator is deliberately transparent.

It does not pretend to perform semantic entailment.

Instead, it uses manually annotated facts to establish the expected behavior of each metric.

This creates a reliable foundation for testing model-based evaluators later.

4.1 Implement F1 scoring

Append the following function to rag_eval.py:

def f1(predicted: set, expected: set) -> float:
    if not predicted and not expected:
        return 1.0

    if not predicted or not expected:
        return 0.0

    overlap = len(predicted & expected)

    return 2 * overlap / (len(predicted) + len(expected))

We will apply this function in two ways.

For factual correctness, we compare complete attribute-value claims.

For our simplified answer-relevance proxy, we compare requested attributes against attributes covered by the response.

These are intentionally different comparisons.

For example:

• ("express_eta", "2 business days") matches the reference fact.

• ("express_eta", "3 business days") does not match it.

• Both cover the same question facet, express_eta.

Consequently, an incorrect delivery time may still receive a perfect facet-relevance score.

That is expected.

Relevance measures whether the response addresses the subject. Correctness measures whether the information is accurate.

4.2 Implement fixed-k retrieval scoring

Append the evaluation function:

def evaluate(trace: Trace) -> dict[str, float | None]:
    top_k = trace.retrieved[:K]

    if len(top_k) != K or len(set(top_k)) != K:
        raise ValueError(
            f"{trace.name}: expected {K} distinct retrieved documents"
        )

    evidence = set().union(
        *(DOCUMENTS[doc_id]["facts"] for doc_id in top_k)
    )

    generated = set(trace.response_claims)
    reference = set(trace.expected_claims)
    relevant = set(trace.relevant_documents)

    unsupported = len(generated - evidence)

    retrieved_relevant = len(
        set(top_k) & relevant
    )

    return {
        "faithfulness": (
            1 - unsupported / len(generated)
            if generated else None
        ),
        "relevance": f1(
            {facet for facet, _ in generated},
            set(trace.requested_facets),
        ),
        "correctness": f1(
            generated,
            reference,
        ),
        "hallucination": (
            unsupported / len(generated)
            if generated else None
        ),
        "precision_at_k": retrieved_relevant / K,
        "recall_at_k": (
            retrieved_relevant / len(relevant)
            if relevant else 1.0
        ),
    }

Let’s examine the implementation carefully.

Evidence construction

We retrieve the structured facts associated with the first k ranked documents.

This models the evidence available to the generator.

The scorer only checks whether a claim appears among those facts.

It does not check whether the source itself is trustworthy.

Faithfulness

We count generated claims absent from retrieved evidence.

If the response contains four claims and one is unsupported:

Faithfulness = 3 / 4 = 0.75

Hallucination rate

We divide unsupported claims by total generated claims.

For a response with no factual claims, both faithfulness and hallucination rate return None.

We should not automatically assign a perfect grounding score to an empty response.

Abstentions require separate evaluation, which we discuss later.

Relevance

We compare requested attributes with generated attributes.

This is a structured proxy, not a replacement for semantic answer relevance.

It cannot reliably judge complex paraphrases, conversational intent, or whether an explanation is sufficiently detailed.

Correctness

We compare complete facts with independently verified reference claims.

An unsupported but correct guess can therefore achieve high correctness while receiving low faithfulness.

Precision@2 and Recall@2

The evaluator always examines the first two documents.

Precision divides by K, not by a variable number of retrieved results.

The explicit input validation ensures that the tutorial’s fixed-k convention cannot silently change between traces.

For production systems that return fewer than k documents, decide on a consistent padding or denominator policy and document it. The present lab intentionally requires two distinct results.

One additional limitation remains: our implementation uses sets, so duplicate claims are collapsed. A production claim-level evaluator may need to retain repeated claim occurrences.

4.3 Add aggregate reporting

Append the reporting function:

def main() -> None:
    fields = (
        "faithfulness",
        "relevance",
        "correctness",
        "hallucination",
        "precision_at_k",
        "recall_at_k",
    )

    results = [evaluate(t) for t in TRACES]

    print(f"case          faith relev correct hallu P@{K}  R@{K}")

    for trace, result in zip(TRACES, results):
        values = " ".join(
            " N/A"
            if result[f] is None
            else f"{result[f]:.2f}"
            for f in fields
        )

        print(f"{trace.name:<13} {values}")

    all_claims = sum(
        len(set(t.response_claims))
        for t in TRACES
    )

    unsupported = sum(
        len(
            set(t.response_claims)
            - set().union(
                *(
                    DOCUMENTS[d]["facts"]
                    for d in t.retrieved[:K]
                )
            )
        )
        for t in TRACES
    )

    print(
        f"Micro unsupported-claim rate: "
        f"{unsupported}/{all_claims} = "
        f"{unsupported / all_claims:.1%}"
    )

    print(
        f"Macro correctness F1: "
        f"{mean(r['correctness'] for r in results):.3f}"
    )


if __name__ == "__main__":
    main()

Run the complete script:

python rag_eval.py

4.4 Examine the verified results

The complete deterministic implementation was executed with Python and produced:

case          faith relev correct hallu P@2  R@2
good_return   1.00 1.00 1.00 0.00 0.50 1.00
extra_claim   0.50 0.67 0.67 0.50 0.50 1.00
lucky_guess   0.00 1.00 1.00 1.00 0.00 0.00
wrong_mirror  1.00 1.00 0.00 0.00 0.00 0.00
off_topic     1.00 0.00 0.00 0.00 0.50 1.00
Micro unsupported-claim rate: 2/6 = 33.3%
Macro correctness F1: 0.533

These results reveal several important patterns.

The good return answer achieves perfect grounding, relevance, and correctness. Retrieval precision is only 0.50 because the system also retrieved an irrelevant packaging document.

The extra-claim answer has 0.50 faithfulness because one of its two claims is unsupported. Its factual correctness F1 is approximately 0.67 because one correct claim is present alongside one incorrect addition.

The lucky guess achieves perfect relevance and correctness despite receiving no supporting evidence. This is unacceptable for applications requiring auditable answers.

The incorrect policy mirror achieves perfect faithfulness but zero correctness.

The off-topic response is fully grounded in one retrieved document but completely fails to answer the question.

This is exactly why RAG quality cannot be represented faithfully by one aggregate score.

4.5 Understand micro versus macro aggregation

The experiment contains six unique generated claims.

Two are unsupported.

Micro hallucination rate = 2 / 6 = 33.3%

Every claim contributes equally.

Macro averaging instead calculates the unsupported-claim rate per response and averages those response-level values.

Consider two answers:

• Answer A contains one unsupported claim.

• Answer B contains 99 supported claims.

Micro hallucination rate = 1 / 100 = 1%

Macro hallucination rate = (100% + 0%) / 2 = 50%

Both calculations are mathematically valid.

They answer different operational questions.

Micro averaging describes the fraction of unsupported claims across all evaluated output.

Macro averaging describes the average unsupported-claim proportion per response.

Production dashboards should identify the aggregation method rather than publishing an unexplained percentage.

5. The cherry on the cake: A confidently wrong answer with perfect faithfulness

The most surprising result is the wrong_mirror case.

The user asks:

How long can I return an unopened product?

The assistant retrieves an unofficial policy mirror stating:

Unopened products may be returned within 90 days.

It responds confidently:

You can return unopened products within 90 days.

Here are the measured scores:

This is a synthetic scenario, not a documented customer-support incident.

Nevertheless, the failure mode is real and generalizable.

A RAG system can faithfully transmit misinformation while receiving a perfect evidence-grounding score.

In this case, the language model has not invented an unsupported claim. It has repeated a claim from an unreliable source.

That distinction matters because generation improvements alone cannot guarantee factual reliability.

How to prevent the wrong-source failure

A trustworthy retrieval architecture should include several safeguards.

• Source authority: Prioritize official documentation over unofficial copies.

• Version control: Associate policies with revision identifiers and effective dates.

• Freshness rules: Exclude outdated policies unless the question explicitly requests historical information.

• Conflict detection: Identify sources that disagree about the same factual attribute.

• Independent verification: Validate high-impact claims against authoritative records.

• Citation auditing: Verify that cited documents support the answer and are authoritative for the question.

Retrieving more documents does not automatically solve the problem.

Adding more context can introduce additional contradictions.

For many production applications, the best improvement is tighter control over document ingestion and retrieval—not a more elaborate generation prompt.

6. Connect the pipeline to modern Ragas metrics

Our deterministic evaluator works because the factual claims have already been annotated.

Production RAG responses are much messier.

They contain paraphrases, compound sentences, uncertainty, implicit assumptions, and terminology that cannot be reduced to predefined attribute-value pairs.

A semantic evaluator needs to perform at least two operations:

1. Extract atomic factual claims from generated responses.

2. Determine whether those claims are supported by evidence or match authoritative references.

Ragas provides model-based metrics for these tasks.

6.1 Current library and model compatibility

For a reproducible integration, we will target Ragas 0.4.3, released January 13, 2026, using its collections-based metric API.

The modern approach creates individual metrics and calls their asynchronous ascore() methods instead of passing collections metrics to the older evaluation workflow.

The Ragas migration documentation describes this architecture.

There are two important compatibility issues to address.

Issue A: Reasoning model parameter conflicts

Ragas 0.4.3 classifies GPT-5 and newer model names as reasoning models.

Its OpenAI parameter adapter sets temperature to 1.0.

However, GPT-6 Luna does not accept temperature at its default reasoning effort.

Sending temperature with the default effort can cause an API error.

The solution for this particular model is to explicitly configure:

• Model: gpt-6-luna

• Reasoning effort: none

• Temperature: 1.0

GPT-6 Luna supports the none reasoning setting, which permits temperature-based sampling.

This compatibility requirement is documented in OpenAI’s GPT-6 model guidance and can be checked against the Ragas 0.4.3 parameter-mapping implementation.

Do not assume that setting a temperature value alone fixes a reasoning-model incompatibility.

The effective reasoning effort also matters.

Issue B: Embedding model currency

The familiar OpenAI text-embedding-3-small model remains documented, but it was introduced in 2024.

For a newer embedding integration, we will use Google’s gemini-embedding-2, released as a stable model in April 2026.

The Gemini Embedding 2 documentation confirms its model identifier and support for text embeddings.

Ragas 0.4.3 provides a Google embedding provider compatible with the modern Google GenAI SDK.

This requires both an OpenAI API key for evaluation generation and a Google API key for embeddings.

6.2 Install the hosted evaluation dependencies

Inside the existing environment:

python -m pip install "ragas==0.4.3" "langchain-community==0.4.1" "openai>=2,<3" "google-genai>=1,<2"

The langchain-community pin matters. Ragas 0.4.3 still imports langchain_community.chat_models.vertexai, which langchain-community 0.4.2 (May 2026) no longer ships. Without the pin, a fresh install fails at import ragas with ModuleNotFoundError. We verified this combination on October 16, 2026.

These version constraints select the tested Ragas interface but do not constitute a fully reproducible transitive dependency lockfile.

For production CI, resolve and pin the complete dependency tree after integration testing.

Set your credentials using your shell’s secret-management mechanism. For a local Bash session:

export OPENAI_API_KEY="your-openai-api-key"
export GOOGLE_API_KEY="your-google-api-key"

Avoid committing keys to source control.

The hosted experiment sends evaluation inputs to external APIs and may incur usage charges.

6.3 Build the model-based evaluator

Create ragas_live.py.

We will use four Ragas metrics:

• Faithfulness

• AnswerRelevancy

• FactualCorrectness

• ContextRecall

The embedding factory must be imported from the current, non-deprecated path:

from ragas.embeddings.base import embedding_factory

Importing it from the package-level embeddings namespace is deprecated in Ragas 0.4.3.

Our implementation uses the modern interface explicitly.

import asyncio
import math
import os

from pydantic import BaseModel
from openai import AsyncOpenAI

from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextRecall,
    FactualCorrectness,
    Faithfulness,
)

from rag_eval import DOCUMENTS, TRACES, K


REFERENCES = {
    "good_return": (
        "Unopened products can be returned within 30 days."
    ),
    "extra_claim": (
        "Express delivery takes 2 business days."
    ),
    "lucky_guess": (
        "Live chat operates weekdays from 09:00 to 17:00."
    ),
    "wrong_mirror": (
        "Unopened products can be returned within 30 days."
    ),
    "off_topic": (
        "Express delivery includes tracking."
    ),
}


class SmokeResult(BaseModel):
    ok: bool


async def main() -> None:
    for key in ("OPENAI_API_KEY", "GOOGLE_API_KEY"):
        if not os.getenv(key):
            raise RuntimeError(f"Missing required variable: {key}")

    client = AsyncOpenAI(timeout=60.0)

    judge = llm_factory(
        "gpt-6-luna",
        client=client,
        reasoning_effort="none",
        temperature=1.0,
        max_tokens=4096,
    )

    embeddings = embedding_factory(
        provider="google",
        model="gemini-embedding-2",
        interface="modern",
    )

    # Exercise structured output, not just model initialization.
    smoke = await judge.agenerate(
        "Return a JSON object whose ok field is true.",
        SmokeResult,
    )

    if not isinstance(smoke, SmokeResult) or not smoke.ok:
        raise RuntimeError("Structured-output smoke test failed")

    # Exercise the real embedding endpoint.
    vector = await embeddings.aembed_text(
        "Express delivery takes two business days."
    )

    if not vector or not all(math.isfinite(x) for x in vector):
        raise RuntimeError("Embedding smoke test failed")

    print("Hosted API smoke tests passed.")
    print(f"Embedding dimensions: {len(vector)}")

    faithfulness = Faithfulness(llm=judge)
    relevance = AnswerRelevancy(
        llm=judge,
        embeddings=embeddings,
        strictness=2,
    )
    correctness = FactualCorrectness(
        llm=judge,
        mode="f1",
    )
    recall = ContextRecall(llm=judge)

    for trace in TRACES:
        contexts = [
            DOCUMENTS[doc_id]["text"]
            for doc_id in trace.retrieved[:K]
        ]

        reference = REFERENCES[trace.name]

        grounded = await faithfulness.ascore(
            user_input=trace.question,
            response=trace.response,
            retrieved_contexts=contexts,
        )

        relevant = await relevance.ascore(
            user_input=trace.question,
            response=trace.response,
        )

        correct = await correctness.ascore(
            response=trace.response,
            reference=reference,
        )

        retrieved = await recall.ascore(
            user_input=trace.question,
            reference=reference,
            retrieved_contexts=contexts,
        )

        scores = {
            "faithfulness": grounded.value,
            "relevance": relevant.value,
            "correctness": correct.value,
            "context_recall": retrieved.value,
        }

        for metric_name, value in scores.items():
            if value is None or not math.isfinite(float(value)):
                raise RuntimeError(
                    f"{trace.name}: invalid {metric_name} score"
                )

        print(
            trace.name,
            " ".join(
                f"{name}={value:.3f}"
                for name, value in scores.items()
            ),
        )


if __name__ == "__main__":
    asyncio.run(main())

This example includes two actual API smoke-test operations before evaluating the dataset.

The first checks whether the evaluator can produce a Pydantic-validated structured response.

The second verifies that the embedding provider returns a nonempty finite vector.

Only after these checks succeed does the program execute the four Ragas metrics.

Execution-status distinction: The standard-library evaluation and CI examples in this tutorial were executed successfully. A live hosted-model execution could not be performed in the credential-free environment used to prepare this tutorial. Therefore, the hosted Ragas code is a documented integration and executable smoke test, not a claimed successful end-to-end API run. Treat it as integration-ready only after the smoke tests and all metric calls pass with your own credentials and resolved dependency versions.

This distinction matters because successful Python syntax validation does not prove provider compatibility.

6.4 Execute the hosted smoke test and evaluation

Run:

python ragas_live.py

A successful initial smoke test prints output with this structure:

Hosted API smoke tests passed.
Embedding dimensions: <positive integer>

The script then prints the metric scores for each test case.

The values are produced by hosted models, so we deliberately do not fabricate example scores.

If the script raises an exception during model initialization, structured-output generation, or embedding, treat the integration as unverified.

Do not silently suppress the error and publish incomplete scores as successful evaluation results.

6.5 Understand what the hosted metrics measure

The semantic metrics will not necessarily reproduce our deterministic scores.

For example, an answer-relevance evaluator might consider the invented free-shipping detail relevant because it discusses the same delivery service.

The deterministic facet metric penalizes that additional attribute.

These scoring differences do not automatically indicate a bug.

They demonstrate that metrics use different definitions and judgment mechanisms.

For automated evaluations, record the metric implementation, configuration, model, and prompt version alongside the scores.

6.6 Why hallucination rate is not an independent Ragas score here

Our deterministic experiment defines grounding hallucination rate as the complement of claim-level faithfulness.

If the semantic faithfulness metric uses the same supported-versus-unsupported classification, a corresponding unsupported-claim proportion can be derived.

But subtracting faithfulness from one does not create an independent detector.

For production applications, preserve claim-level labels distinguishing:

• Supported claims.

• Claims contradicted by retrieved evidence.

• Claims lacking sufficient evidence.

• Claims contradicted by authoritative references.

This classification produces more useful diagnostics than a single percentage.

7. Calibrate automated judges against human labels

Using a language model as an evaluator introduces another potential source of error.

The judge may misunderstand evidence, incorrectly combine claims, overlook contradictions, or produce unstable scores.

Evaluation quality therefore needs its own evaluation process.

7.1 Create a human-labeled calibration set

Construct a representative sample of question-response-evidence triples.

Have reviewers annotate:

• Atomic factual claims.

• Supporting evidence spans.

• Contradicting evidence spans.

• Missing information.

• Relevance to the original question.

• Correctness against authoritative references.

Provide a written annotation policy.

For example, reviewers must know whether a fact that is true but absent from retrieval should count as unsupported.

Without explicit rules, disagreements may reflect inconsistent definitions rather than evaluator failures.

7.2 Measure unsupported-claim detection

Suppose human reviewers annotate 200 claims as either supported or unsupported.

Compare the automated evaluator’s predictions with human labels.

Calculate:

Unsupported-claim precision = correctly flagged unsupported claims / all claims flagged unsupported

Unsupported-claim recall = correctly flagged unsupported claims / all truly unsupported claims

Also report false positives and false negatives.

False negatives are especially important when the evaluator serves as a quality gate: an unsupported claim that escapes detection may be delivered to users.

A single agreement percentage can hide a serious imbalance between these error types.

7.3 Test evaluator stability

Repeat a subset of evaluations and record:

• Evaluator model identifier.

• Exact model snapshot, when available.

• Evaluation prompts.

• Claim extraction instructions.

• Sampling and reasoning configurations.

• Document revisions.

• Generated judgments.

• Evidence spans used for each decision.

A metric that varies unpredictably across repeated executions should not be used as a strict deployment gate without calibration.

7.4 Defend the evaluator against prompt injection

Retrieved documents are untrusted inputs.

A malicious document might include instructions telling the generator or evaluator to ignore previous instructions and approve an unsupported answer.

Treat such content as data, not instructions.

Evaluation prompts should explicitly distinguish evidence from evaluator directives.

Additionally:

• Validate structured outputs.

• Avoid granting evaluation models unnecessary tools.

• Use clear evidence boundaries.

• Include adversarial instructions in calibration examples.

• Review unexpected changes in evaluator behavior.

A model-based judge should not become a second attack surface.

8. Make the evaluation dataset production-representative

Our five-case benchmark is useful for understanding metrics.

It is not sufficient to establish production reliability.

A serious benchmark should cover the actual distribution of user questions and failure modes.

8.1 Include diverse question types

Each category should include both successful and failing examples.

A benchmark containing only answerable questions may encourage a system to generate an answer even when sufficient evidence does not exist.

8.2 Evaluate abstention explicitly

Suppose a user asks:

What is the warranty period for product X?

The knowledge base contains no warranty information.

The assistant responds:

I cannot verify the warranty period from the available documentation.

This may be exactly the right answer.

A naive claim-based faithfulness metric may find no factual claims and be unable to calculate a useful score.

Instead, evaluate:

• Answerability accuracy: Does the system recognize when evidence is sufficient?

• Abstention precision: How often is abstention justified?

• Abstention recall: How often does the assistant abstain when information is insufficient?

• Answer coverage: What fraction of questions receive substantive answers?

These metrics expose an important trade-off.

A system that refuses every question may avoid unsupported factual claims while being practically useless.

8.3 Maintain independent evaluation splits

Separate your examples into several groups:

• Development examples for prompt and retrieval tuning.

• Regression examples containing previously identified failures.

• Held-out evaluation cases for unbiased model comparison.

• Fresh production-like samples for detecting distribution shifts.

Avoid repeatedly optimizing against the same held-out evaluation set.

Over time, those examples become part of the development process and stop providing independent evidence of generalization.

9. Measure uncertainty when comparing systems

Suppose two RAG versions achieve average faithfulness scores of 0.88 and 0.90.

The second version appears better.

But is that improvement statistically meaningful?

Not necessarily.

The difference could be caused by sample composition, evaluator noise, or random variation.

9.1 Prefer paired comparisons

Evaluate both systems on the same questions.

For every question, compute:

Difference = score from version B − score from version A

Then examine the distribution of differences.

A model might improve significantly on direct lookups while regressing on multi-hop questions.

A single average could hide that regression.

9.2 Estimate confidence intervals using bootstrapping

We can illustrate bootstrap uncertainty using the deterministic benchmark.

Create bootstrap_eval.py:

from random import Random
from statistics import mean

from rag_eval import TRACES, evaluate

rng = Random(42)

scores = [
    evaluate(trace)["correctness"]
    for trace in TRACES
]

bootstrap_means = []

for _ in range(2000):
    sample = rng.choices(scores, k=len(scores))
    bootstrap_means.append(mean(sample))

bootstrap_means.sort()

lower_index = int(0.025 * len(bootstrap_means))
upper_index = int(0.975 * len(bootstrap_means))

lower = bootstrap_means[lower_index]
upper = bootstrap_means[upper_index]

print(f"Observed mean: {mean(scores):.3f}")
print(f"Bootstrap 95% interval: [{lower:.3f}, {upper:.3f}]")

Run:

python bootstrap_eval.py

The procedure repeatedly samples evaluation examples with replacement and calculates the average correctness score.

The resulting percentile interval estimates uncertainty under the assumption that the observed examples reasonably represent the target population.

With only five deliberately constructed cases, the interval is illustrative—not a defensible estimate of production uncertainty.

For production analysis, collect a much larger representative sample.

When comparing two versions, resample paired per-question differences rather than independently bootstrapping each system.

Also distinguish statistical uncertainty from evaluator disagreement. A narrow confidence interval is not reassuring if the evaluator systematically labels incorrect claims as correct.

10. Separate regression tests from candidate-system quality gates

One common CI mistake is applying strict quality thresholds directly to a benchmark containing deliberately failing fixtures.

Our five-case dataset intentionally includes bad responses.

If we require every original trace to achieve faithfulness above 0.95, the test suite will always fail.

That is not a useful deployment acceptance test.

We need two separate mechanisms:

1. Known-failure regression tests: Verify that evaluation metrics still detect established failure patterns.

2. Candidate-system quality gates: Verify that a new system produces acceptable answers and retrieval results.

These should not be confused.

10.1 Test known failure patterns

Create regression_tests.py:

from rag_eval import TRACES, evaluate

results = {
    trace.name: evaluate(trace)
    for trace in TRACES
}

assert results["good_return"]["faithfulness"] == 1.0

assert results["extra_claim"]["hallucination"] == 0.5

assert results["lucky_guess"]["correctness"] == 1.0
assert results["lucky_guess"]["faithfulness"] == 0.0

assert results["wrong_mirror"]["faithfulness"] == 1.0
assert results["wrong_mirror"]["correctness"] == 0.0
assert results["wrong_mirror"]["precision_at_k"] == 0.0

assert results["off_topic"]["faithfulness"] == 1.0
assert results["off_topic"]["relevance"] == 0.0

print("Five known-case regression tests passed.")

Run:

python regression_tests.py

The verified result is:

Five known-case regression tests passed.

Notice the counterintuitive behavior.

The test passes because the evaluator correctly recognizes that the deliberately incorrect answer has perfect grounding but zero factual correctness.

Passing this regression test does not mean that the bad response is acceptable.

It means our evaluation logic still identifies its expected characteristics.

10.2 Define a candidate-system quality gate

Next, we need a separate file containing the actual outputs of a candidate RAG system.

The file will contain:

• Benchmark case identifier.

• Generated response.

• Retrieved document IDs.

• Annotated claims extracted from the candidate response.

The candidate supplies outputs, not ground-truth references.

Ground-truth claims and relevant-document labels remain in the benchmark.

Create candidate_gate.py:

import json
import sys

from dataclasses import replace
from pathlib import Path

from rag_eval import TRACES, evaluate


if len(sys.argv) != 2:
    raise SystemExit(
        "Usage: python candidate_gate.py candidate_traces.json"
    )

gold = {
    trace.name: trace
    for trace in TRACES
}

rows = json.loads(
    Path(sys.argv[1]).read_text(encoding="utf-8")
)

if not isinstance(rows, list):
    raise SystemExit(
        "Candidate file must contain a JSON list"
    )

seen = [row["name"] for row in rows]

if len(seen) != len(set(seen)) or set(seen) != set(gold):
    raise SystemExit(
        "Exactly one candidate result per benchmark case is required"
    )

failures = []

for row in rows:
    reference = gold[row["name"]]

    trace = replace(
        reference,
        response=row["response"],
        retrieved=tuple(row["retrieved"]),
        response_claims=tuple(
            tuple(claim)
            for claim in row["response_claims"]
        ),
    )

    scores = evaluate(trace)

    for metric, minimum in (
        ("faithfulness", 0.95),
        ("relevance", 0.95),
        ("correctness", 0.95),
        ("recall_at_k", 1.0),
    ):
        value = scores[metric]

        if value is None or value < minimum:
            failures.append(
                f"{trace.name}: "
                f"{metric}={value}, minimum={minimum}"
            )

if failures:
    print("Candidate quality gate failed:")

    for failure in failures:
        print("-", failure)

    raise SystemExit(1)

print(
    f"Candidate quality gate passed for {len(rows)} cases."
)

This gate applies to candidate outputs rather than the intentionally failing reference fixtures.

The thresholds are demonstrations, not universal industry standards.

The requirements are deliberately strict because this synthetic lab has simple, fully answerable questions.

10.3 Generate a passing synthetic candidate for testing

Before connecting a real RAG system, verify that the gate accepts correctly formed candidate traces.

Create make_demo_candidate.py:

import json
from pathlib import Path

from rag_eval import TRACES


ANSWERS = {
    "good_return": (
        "Unopened products can be returned within 30 days."
    ),
    "extra_claim": (
        "Express delivery takes 2 business days."
    ),
    "lucky_guess": (
        "Live chat operates weekdays from 09:00 to 17:00."
    ),
    "wrong_mirror": (
        "Unopened products can be returned within 30 days."
    ),
    "off_topic": (
        "Express delivery includes tracking."
    ),
}

rows = [
    {
        "name": t.name,
        "response": ANSWERS[t.name],
        "retrieved": [
            t.relevant_documents[0],
            "packaging",
        ],
        "response_claims": [
            list(claim)
            for claim in t.expected_claims
        ],
    }
    for t in TRACES
]

Path("candidate_traces.json").write_text(
    json.dumps(rows, indent=2),
    encoding="utf-8",
)

print("Wrote synthetic candidate_traces.json")

This script constructs deliberately corrected candidate traces from known benchmark labels.

It is a quality-gate fixture, not a real retrieval or generation implementation.

Run:

python make_demo_candidate.py
python candidate_gate.py candidate_traces.json

The verified output is:

Wrote synthetic candidate_traces.json
Candidate quality gate passed for 5 cases.

Now the CI architecture distinguishes three things:

• The evaluator itself works.

• Deliberate known failures are detected.

• A candidate system can satisfy the acceptance criteria.

To use this gate in production, replace the synthetic candidate file with the actual responses and retrieved document IDs produced by your RAG pipeline.

Critically, candidate claim annotations must be generated by a trusted extraction process or reviewed independently. Otherwise, a system could appear perfect by omitting inconvenient claims from its annotation records.

For natural-language candidates, the deterministic attribute matcher should also be replaced or supplemented with calibrated semantic claim verification.

10.4 Run the complete deterministic CI suite

The main checks can be executed sequentially:

python rag_eval.py
python regression_tests.py
python make_demo_candidate.py
python candidate_gate.py candidate_traces.json

All four commands were exercised for the deterministic implementation.

In a real CI pipeline, run the regression tests for every change. Generate candidate traces from the actual candidate system rather than invoking the synthetic fixture generator as a substitute for evaluation.

10.5 Use risk-based deployment thresholds

A production acceptance gate should consider more than minimum average scores.

Useful checks include:

• Minimum faithfulness for answerable cases.

• Maximum unsupported-claim rate.

• Minimum factual correctness.

• Minimum Recall@k.

• Maximum error rates for critical question categories.

• Acceptable abstention behavior.

• Maximum latency and cost.

• No regressions on designated high-severity cases.

For example, a return-policy assistant may have a hard requirement that active return windows match the current authoritative policy.

That rule should be represented by a dedicated regression test.

Averages should not allow one important factual failure to disappear among hundreds of successful responses.

11. Diagnose failures using metric combinations

Individual metrics become especially useful when examined together.

Consider the following diagnostic table.

These are diagnostic clues rather than definitive explanations.

For example, a semantic evaluator might incorrectly reject a faithful paraphrase.

Similarly, a low retrieval-recall score may reflect incomplete reference annotations rather than a defective retriever.

Inspect the underlying traces before changing prompts, retrieval algorithms, or model configurations.

11.1 Keep retrieval and generation experiments separate

If retrieval recall falls, consider:

• Document ingestion completeness.

• Chunk boundaries.

• Search indexing.

• Query transformation.

• Metadata filters.

• Hybrid lexical and vector retrieval.

• Reranking quality.

If retrieval remains strong but faithfulness falls, investigate:

• Generator instructions.

• Context truncation.

• Conflicting passages.

• Long-context distraction.

• Unsupported elaboration.

• Failure to abstain when evidence is ambiguous.

Changing multiple components simultaneously makes root-cause analysis harder.

Whenever practical, run controlled experiments that modify one major variable at a time.

12. Build a production evaluation architecture

A mature evaluation system should preserve enough information to explain and reproduce scoring decisions.

A practical architecture looks like this:

Production or benchmark question
            |
            v
       RAG retriever
            |
            v
   Ranked document chunks
            |
            v
       RAG generator
            |
            v
      Generated answer
            |
            v
   Claim extraction pipeline
            |
            v
   +----------------------+
   | Evidence grounding   |
   | Answer relevance     |
   | Factual correctness  |
   | Retrieval evaluation |
   +----------------------+
            |
            v
     Evaluation records
            |
            v
   Regression and quality gates
            |
            v
      Monitoring dashboard

For each evaluated request, record:

• Question and conversation context.

• Retrieved document identifiers.

• Retrieval ranking positions.

• Source revisions and effective dates.

• Generated response.

• Citations.

• Atomic extracted claims.

• Supporting or contradicting evidence.

• Reference answer, when available.

• Individual metric scores.

• Evaluator configuration and model versions.

• Latency, token usage, and cost.

Claim-level evidence is particularly valuable.

Instead of observing that average faithfulness declined from 0.94 to 0.87, engineers can determine exactly which statements became unsupported.

They can then investigate the associated retrieval, source, prompt, or generation changes.

12.1 Monitor production drift

Offline evaluation captures known failure patterns.

Production monitoring detects new ones.

Useful distributions to monitor include:

• Question categories.

• Document age.

• Source authority.

• Retrieval relevance.

• Answer length.

• Unsupported-claim frequency.

• Abstention frequency.

• Source conflicts.

• User-reported factual errors.

A sudden decline in retrieval recall after reindexing may indicate a document-ingestion problem.

A rise in unsupported claims following a prompt change may suggest that the generator has started adding unnecessary details.

An increase in conflicting policy documents may indicate that obsolete material has entered the active index.

Link these measurements to identifiable deployment versions.

12.2 Track latency and cost alongside quality

Accuracy alone is insufficient for operating a production RAG system.

A larger retrieval context might improve grounding but double inference cost.

A more capable evaluator might improve claim verification while making every test execution substantially slower.

Track the following dimensions together.

The objective is not to maximize a single metric.

It is to satisfy the application’s quality, risk, latency, and cost requirements simultaneously.

12.3 Establish a continuous improvement loop

A reliable evaluation process follows a repeating cycle:

1. Capture representative production traces.

2. Detect suspicious outputs through automated evaluation and user feedback.

3. Review selected failures manually.

4. Add confirmed problems to the regression dataset.

5. Modify the relevant retrieval or generation component.

6. Compare the updated system against its predecessor.

7. Deploy only after applicable acceptance gates pass.

8. Continue monitoring for new failure patterns.

This makes evaluation part of engineering rather than an occasional benchmarking exercise.

13. Common mistakes that undermine RAG evaluation

Even sophisticated evaluation pipelines can produce misleading results.

Mistake 1: Reporting only one quality score

A system can be perfectly grounded in inaccurate documents.

Report faithfulness and independent factual correctness separately.

Mistake 2: Assuming citations prove truth

A citation may point to a source that is irrelevant, outdated, contradicted, or unauthoritative.

Citation presence is not citation validity.

Citation validity is not factual correctness.

Mistake 3: Calling every unsupported claim false

Evidence absence and factual falsehood are different conditions.

Use separate labels whenever possible.

Mistake 4: Treating embedding similarity as entailment

Statements about 30-day and 90-day return windows are semantically similar but factually contradictory.

Cosine similarity cannot reliably establish logical consistency.

Mistake 5: Using a variable retrieval cutoff while reporting Precision@k

If k changes across examples, the resulting scores may no longer be comparable under the intended definition.

Use a fixed cutoff and document behavior for short result lists.

Mistake 6: Trusting an uncalibrated language-model judge

A model-generated score is not automatically ground truth.

Compare automated decisions against independently reviewed labels.

Mistake 7: Ignoring model and SDK compatibility

A package can install successfully while failing on its first API request.

Verify model initialization, structured output, supported request parameters, and embedding responses.

Pin dependencies and run integration smoke tests in the intended deployment environment.

Mistake 8: Confusing evaluation regression tests with candidate acceptance tests

Deliberately failing fixtures should remain failing examples.

Their associated regression tests should verify that the evaluator detects those failures.

Deployment gates should assess candidate-system behavior against acceptance criteria.

Mistake 9: Ignoring abstention

Questions without sufficient evidence are part of real-world workloads.

Evaluate whether the system knows when not to answer.

Mistake 10: Publishing scores without uncertainty or evaluation provenance

A score without its dataset, evaluator configuration, and aggregation method is difficult to interpret or reproduce.

Store enough metadata to reconstruct the result.

14. Production readiness checklist

Before deploying a RAG evaluation pipeline, verify the following requirements.

• The benchmark contains representative questions and independently verified references.
• Claim-level faithfulness is evaluated against retrieved evidence.
• Answer relevance is measured separately from correctness.
• Unsupported-claim and factual-error rates use explicit definitions.
• Retrieval precision and recall are evaluated at a fixed k.
• Source conflicts, outdated documents, and distractors are represented.
• Questions without sufficient evidence are included.
• Abstention behavior is evaluated independently.
• Model-based evaluators are calibrated against human labels.
• Package versions, models, prompts, and source revisions are recorded.
• Known-failure regression tests are separate from candidate acceptance gates.
• Hosted API smoke tests verify structured output and embeddings before evaluation.
• Candidate systems are compared on paired examples.
• Quality thresholds reflect error severity and uncertainty.
• Latency, reliability, and cost are tracked alongside answer quality.
• Confirmed production failures are added to the regression suite.

Completing this checklist does not guarantee error-free RAG.

It establishes the foundation for detecting, explaining, and systematically reducing errors.

15. Conclusion: Evaluate the answer, the evidence, and the truth

Reliable RAG evaluation is not simply about checking whether an answer sounds convincing.

A trustworthy evaluation process must answer three independent questions:

Is the answer relevant to what the user asked?

Is the answer supported by the documents the system retrieved?

Is the answer factually correct according to authoritative information?

Retrieval precision and recall explain whether the system found the necessary information.

Faithfulness measures whether the generator respected that evidence.

Correctness determines whether the generated claims match independently verified truth.

Hallucination metrics quantify unsupported assertions, but their definitions must be precise enough to distinguish missing evidence from genuine factual errors.

Our synthetic experiment showed why these distinctions matter.

The most striking case was a confident, incorrect 90-day return policy that achieved perfect faithfulness because it repeated a false claim from an unreliable document.

That is the central lesson:

A system can be perfectly grounded in the wrong information.

The deterministic evaluation harness provides a reproducible starting point for understanding these metrics. The Ragas integration demonstrates how to extend the same concepts toward semantic evaluation, with explicit model compatibility requirements and API smoke tests.

The regression suite and candidate quality gate show how to turn evaluations into automated engineering controls without confusing expected failures with acceptable behavior.

Your next step

Run the deterministic five-case benchmark locally. Confirm that the known-failure assertions pass and that the corrected synthetic candidate passes the acceptance gate.

Then connect your own RAG pipeline and deliberately introduce three defects:

• An unsupported factual claim.

• An irrelevant retrieved document.

• An outdated authoritative-looking source.

Examine which metric detects each defect, which metrics miss it, and what evidence explains the result.

Once the hosted integration passes its credential-gated smoke tests, compare its semantic judgments against your human-labeled examples. Do not promote model-generated scores into production gates until you understand their failure modes.

Finally, convert every confirmed failure into a permanent regression test.

A trustworthy RAG system is not one that never makes mistakes. It is one whose mistakes can be detected, measured, explained, and systematically reduced.