When a RAG answer is wrong, changing the prompt should not be the only first response. Check whether the corpus contains the answer, whether necessary evidence reached the context, and whether generation followed that evidence. Otherwise corpus gaps, retrieval misses and generation failures become one vague model problem.

Locate the failure in three stages

RAG combines retrieved external text with generation. The original 2020 paper studies a particular retriever-generator combination; arbitrary retrieval-and-prompt systems do not inherit its evaluation results.

Create manually reviewed answers and necessary evidence sets. If the corpus lacks that evidence, the system should acknowledge insufficient support: retrieval recall cannot repair absent material. If evidence exists but is missed, investigate retrieval and filtering.

Track retrieval and fidelity separately

Use four fictional questions: evidence retrieved and a correct answer; available evidence missed; evidence retrieved but a wrong answer; and no answer in the corpus. Three are answerable, and two retrieve the needed evidence. Retrieval recall in this example is 2/3.

This teaching example simplifies complete necessary-evidence retrieval to a Boolean. Real questions may require multiple passages. Record per-question evidence coverage, filtering and context truncation. Do not put unanswerable questions into the same recall denominator.

Related documents are not sufficient evidence

Suppose fictional product documentation says that export is supported but specifies no format. Claiming CSV export is unsupported by that passage. A valid link does not prove every sentence in an answer.

Break answers into checkable factual claims and locate supporting passages for each. Separate incorrect inference from citing the wrong version. When sources conflict, retain dates and versions rather than selecting whichever sounds more confident.

def diagnose(answerable, retrieved, faithful):
    if not answerable:
        return "no corpus evidence"
    if not retrieved:
        return "retrieval miss"
    return "grounded answer" if faithful else "generation failure"

cases = [(True,True,True),(True,False,False),
         (True,True,False),(False,False,False)]
assert [diagnose(*c) for c in cases] == [
    "grounded answer", "retrieval miss",
    "generation failure", "no corpus evidence"]
answerable_cases = [c for c in cases if c[0]]
retrieval_recall = sum(c[1] for c in answerable_cases)/len(answerable_cases)
assert retrieval_recall == 2/3
print(round(retrieval_recall,6))

Build a reproducible small evaluation

The code was executed to verify four diagnostic paths and the illustrative 2/3 calculation. It runs no retriever or LLM. Production evaluation also needs a fixed question set, corpus snapshot, settings, model version and annotation criteria; retain failure examples rather than only an aggregate score.

Change one interpretable factor at a time: compare prompts with fixed retrieval results, or compare retrieval configurations with a fixed generator. See CrossEncoder reranking and document chunking for ranking and passage boundaries.

中文版

Sources

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Prepared on October 12, 2026; the article date retains its planned slot.