Two RAG reports both say Recall@2, but one reports 44.44% and the other 40%. Neither calculation necessarily contains an arithmetic mistake. Averaging query-level recall and dividing pooled hit counts by pooled relevance counts assign different weights. Fix the denominator, labeling unit, and measurement stage before deciding whether a change improved retrieval.

One reproducible dataset is enough to expose the difference. The local scorer below rejects empty judgments explicitly and distinguishes finding one relevant item from covering all labeled items. It extends the site’s RAG evaluation and failure diagnosis article without treating offline recall as answer accuracy. 阅读中文版.

Fix the denominator for one query first

The information retrieval textbook defines recall as the fraction of relevant items retrieved. For a cutoff k, let G be the reviewed relevant set and T the set of distinct items in the first k results. This article uses:

G is the relevance set in this evaluation, not an omniscient list of every useful source in existence. If judgments cover a fixed corpus or only part of it, describe that boundary. A score of 0.8 initially means that four fifths of the currently labeled relevant items were found. It does not automatically mean the knowledge base can answer four fifths of user questions.

The cutoff limits the ranked prefix being inspected; the denominator remains the size of the relevant set, not k. Dividing hits by k describes a different quantity. A result between zero and one is not enough to identify which metric was computed.

Calculate two averages on the same results

The following three queries have reviewed, nonempty relevant sets. All identifiers and rankings are fictional, and each query is scored at k=2. A has three relevant items and retrieves one; B has one and retrieves it; C has one and misses it.

The query-level mean is (1/3+1+0)/3=4/9, approximately 44.44%. Pooling counts gives (1+1+0)/(3+1+1)=2/5, or 40%. Nothing about the retrieval output changed. Only the weighting changed.

Macro averaging weights each query equally. Pooling gives a query with more labeled relevant items greater influence. Neither is universally correct without considering the intended interpretation. Equal query weights are easy to explain for a typical-query experience; pooled coverage can be useful when the business quantity is relevant-item coverage. Keep the choice fixed across comparisons.

When relevant items outnumber k, perfect recall is unreachable

With distinct results, Recall@k cannot exceed min(k,|G|)/|G|. Query A has three relevant items but only two slots. Even if both top results are relevant, its recall is at most 2/3 and all-hit remains zero. The code adds an assertion for that ideal-ranking boundary.

This is not an excuse for missing evidence. It explains why the same cutoff can impose different difficulty across queries. If the distribution of relevant-set sizes changes between two query batches, their scores can change too. Preserve that distribution or compare paired results on the same queries rather than interpreting every aggregate drop as retriever regression.

Pooled recall is also a weighted mean of query recall, with weight |Gᵢ|/Σ|Gⱼ| for query i. Stating those weights explains why A contributes 60% under pooling but only one third under equal query averaging. It is another interpretation of the same retrieval experiment, not a new run.

One hit and complete coverage answer different questions

A and B each retrieve at least one relevant item, so the fraction of queries with any hit is 2/3. Only B retrieves its entire labeled relevant set, so complete coverage occurs for 1/3 of queries. These differ from both 44.44% and 40% because they ask different questions.

This article labels those additional statistics any-hit and all-hit rather than assuming that every tool uses “hit rate” identically. A question requiring several pieces of evidence is not necessarily answerable after one relevant passage is found.

Complete labeled-set coverage is not automatically the same as having sufficient answer evidence either. Labels may include alternative sources, all of which need not be retrieved. If the requirement is A and B together, or either A or B, encode that evidence condition separately. Do not silently reinterpret an ordinary relevance set as a complete logical specification.

A runnable scoring reference

The code expects stable identifiers and rejects duplicates so that the upstream evaluation process must choose its deduplication policy explicitly. It also rejects empty relevance sets, avoiding division by zero or treating uncertainty as failure. Only reviewed, nonempty sets are scored; no retriever or language model is called.

import math

def score_query(ranked, gold, k):
    if type(k) is not int or k <= 0:
        raise ValueError("k must be a positive integer")
    if len(ranked) != len(set(ranked)):
        raise ValueError("deduplicate stable IDs before scoring")
    gold = set(gold)
    if not gold:
        raise ValueError("nonempty reviewed relevance labels required")
    hits = len(set(ranked[:k]) & gold)
    return {
        "hits": hits,
        "relevant": len(gold),
        "recall": hits / len(gold),
        "any_hit": int(hits > 0),
        "all_hit": int(hits == len(gold)),
    }

# All identifiers and rankings are invented teaching data.
cases = [
    (["a", "x", "b", "c"], {"a", "b", "c"}),
    (["x", "d"], {"d"}),
    (["x", "y"], {"e"}),
]
rows = [score_query(ranked, gold, 2) for ranked, gold in cases]
macro = sum(r["recall"] for r in rows) / len(rows)
pooled = sum(r["hits"] for r in rows) / sum(r["relevant"] for r in rows)
any_hit = sum(r["any_hit"] for r in rows) / len(rows)
all_hit = sum(r["all_hit"] for r in rows) / len(rows)
assert math.isclose(macro, 4 / 9)
assert math.isclose(pooled, 2 / 5)
assert math.isclose(any_hit, 2 / 3)
assert math.isclose(all_hit, 1 / 3)
assert score_query([], {"a"}, 2)["recall"] == 0
assert score_query(["a"], {"a", "b"}, 5)["recall"] == 0.5
for ranked, gold, k in [(["a", "a"], {"a"}, 2),
                         (["a"], set(), 2),
                         (["a"], {"a"}, 0)]:
    try:
        score_query(ranked, gold, k)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid evaluation input accepted")
print("macro=%.6f pooled=%.6f any=%.6f all=%.6f" %
      (macro, pooled, any_hit, all_hit))

best_at_two = score_query(["a", "b"], {"a", "b", "c"}, 2)
assert math.isclose(best_at_two["recall"], 2 / 3)
assert best_at_two["all_hit"] == 0
weights = [r["relevant"] / sum(v["relevant"] for v in rows) for r in rows]
assert math.isclose(sum(w * r["recall"] for w, r in zip(weights, rows)), pooled)

The output is macro=0.444444, pooled=0.400000, any=0.666667, all=0.333333. Assertions also cover empty results, fewer results than k, duplicate identifiers, empty judgments, and an invalid cutoff. All code was executed locally. These are exact calculations on teaching data, not benchmark scores for a real RAG system.

If an evaluation batch contains no scorable queries, its mean is undefined too. Return an explicit status and report inclusion and exclusion counts with reasons rather than filling in zero. A dashboard that always displays a number can conceal a broken denominator.

Empty judgments can mean different things

A query may not have been reviewed yet, so its relevant items are unknown. Alternatively, reviewers may have confirmed that the fixed corpus contains no answer. Both situations can appear as an empty list, but they should not enter the same statistical branch. The first needs labeling; the second may belong in an abstention or no-answer evaluation.

Store review status separately from the item list. Unreviewed, reviewed with relevant items, and reviewed with no answer should be distinguishable. Rejecting an empty set in this function leaves that decision to the surrounding evaluation process. It does not prescribe deleting every no-answer query.

Incomplete labels can also disadvantage a new system. Items frequently found by the old method may already be judged, while newly surfaced material remains unreviewed. Treating all of it as irrelevant can distort the comparison. Preserve an unjudged state, inspect new results, and compare systems against the same updated judgment snapshot rather than arguing only over the higher aggregate.

Align the measurement stage

A pipeline might retrieve 100 candidates, rerank ten, and fit only five passages into a length-limited model context. A report on the candidate pool and a report on final context do not measure the same output, even if their headings use the same metric name.

Record the actual ordered identifiers at each stage and score them against the same labels at the same cutoff. Missing candidate evidence points toward retrieval or filtering. Evidence present in candidates but absent from final context points toward ranking, deduplication, or the context budget. One final aggregate can otherwise combine several different failures.

When only order changes and the candidate set stays fixed, full-pool set recall stays fixed. Recall at a smaller cutoff can still change. The site’s Cross-Encoder reranking article explains the mechanism; the concern here is keeping the measurement definition stable.

A chunking change can invalidate an old denominator

Splitting one document into more passages changes the retrieval unit. Document-level labels cannot be compared directly with passage-level counts without an explicit mapping. State whether each identifier represents a document, a chunk, or an evidence point, and keep that unit fixed within a comparison.

Overlapping passages can create apparent repeated coverage. Three nearly identical chunks should not automatically become three independent evidence items. Aggregate by stable document identifiers or maintain reviewed chunk-level judgments according to the question being evaluated. Apply and version that mapping before scoring.

The cutoff also has assumptions. For nested prefixes of one fixed ranking and a fixed relevance set, increasing k cannot reduce set recall. If each cutoff triggers a different retrieval strategy, the outputs need not be nested prefixes, and the property cannot simply be assumed. Higher recall also does not automatically offset latency, redundant context, or changes in generation quality.

Make the next report comparable

Retain the query set and corpus snapshot, labeling status and unit, measurement stage, cutoff, deduplication rule, per-query results, and aggregation method. Alongside the headline score, show how many queries were scorable and why others were excluded. Readers should be able to reconstruct the denominator.

Inspect groups when a method improves only certain questions, while preserving paired results on the same examples. Do not switch to a more flattering aggregate or silently remove difficult cases. Statistical uncertainty is a separate question; exact arithmetic on three teaching queries is not a stable estimate of population performance.

Recall helps locate which known relevant items failed to reach a specified pipeline stage. Establish that meaning first, then consider evidence sufficiency, answer faithfulness, and practical cost when deciding which change is worth keeping.

References

Introduction to Information Retrieval: set-based recall; evaluating ranked results at a fixed cutoff. The three-query data, aggregation derivation, graphics, and code are original teaching examples. No live retrieval experiment was performed.