An NDCG of one inside a candidate set does not establish that retrieval found the best documents. If the ideal ranking is calculated only from returned candidates, a missed highly relevant document disappears from the denominator too. A good ordering of surviving candidates can then be mistaken for a good end-to-end retrieval result. Fix the evaluation universe before interpreting the score.

Four fictional documents are enough to compute NDCG@2 by hand and make the same result score either one or approximately 0.617. We then inspect zero-IDCG queries, averaging and duplicate documents. The aim is a reproducible RAG retrieval evaluation, not a claim that an attractive decimal proves answer quality. 中文版.

Specify the document universe behind the ideal ranking

The scikit-learn 1.7.2 NDCG documentation describes normalization by the gain of the best possible ranking. The decisive question is what makes a document eligible for that ideal ranking: a fixed evaluation universe or the candidates surviving a particular retrieval run? Both can answer useful questions, but they must not be silently treated as interchangeable.

Fix the judged set J={A,B,C,D}, with relevance grades 3,2,1,0. These are teaching labels, not measured data. Each query has its own relevance judgments. The system returns ordered document IDs; the evaluator obtains grades from an independent truth table. A model’s similarity score is not a substitute for judged relevance.

This example uses linear gain, gain(r)=r, and one-based position discount log₂(i+1): DCG@K=Σᵢ₌₁ᴷrᵢ/log₂(i+1). Other conventions use gain 2ʳ−1, emphasizing higher grades more strongly. Their values should not be compared directly with the linear version. Recording only the label NDCG@10 leaves essential evaluation choices unspecified.

The same output can score 0.617 or one

Suppose the system returns [B,C]. Position one has discount denominator one and position two has log₂3. Thus DCG@2=2+1/log₂3, approximately 2.630930. Over the fixed full judged set, the ideal order is [A,B], giving IDCG@2=3+2/log₂3, approximately 4.261860. The ratio is approximately 0.617320.

Now remove unreturned A and D before constructing the ideal ranking. Among the remaining candidates [B,C], the ideal order is also [B,C]. The numerator stays unchanged while the denominator falls to the same 2.630930, producing NDCG one. Retrieval did not improve and no document moved. Only the evaluation question changed.

A candidate-only metric can ask whether a reranker approaches the best ordering available within those candidates. An end-to-end metric must also reflect whether better documents were retrieved. If two retrievers each define their ideal universe using their own returned candidates, even a weaker candidate set can normalize itself to one. That is not evidence that the two retrieval systems are equally effective.

A fixed denominator exposes missed retrieval opportunities

Within fixed J, returning [A,B] gives NDCG@2 one. Returning only [B] gives zero contribution at the missing second position, while the ideal denominator still comes from J. Do not automatically shrink IDCG to the number of returned results. The cutoff K and the length of a system’s output are different inputs. Missing positions have zero gain here; the evaluator does not append unreturned documents in an invented low-score order.

Even without ties, do not rebuild the judged universe independently from each system’s output. Fix it before comparison and bind judgments to query and document IDs. On a rerun, replacing only the returned ordering while keeping truth unchanged preserves the evaluation question.

If K exceeds the judged set’s size, the ideal calculation uses the documents available in that set. Nonexistent ideal items and missing result positions contribute no positive gain. The implementation rejects duplicate IDs so that a single highly relevant document cannot earn credit repeatedly at multiple positions. Deduplication by document, page or chunk should match the entity defined by the relevance judgments.

Real RAG evaluations often judge only part of a corpus. They cannot pretend to know the ideal ranking of every unjudged document. Describe how the judgment pool was built, which items have labels and how unjudged results are treated. This evaluator rejects an unknown ID rather than silently equating absent judgment with confirmed irrelevance, forcing the caller to choose a policy explicitly.

If an evaluation protocol assigns zero to unjudged documents, keep the same pool and protocol across systems and explain their coverage limits. A system finding useful documents outside the judged pool may not receive appropriate credit. That is a limitation of the available evidence, not a license to invent a precise ranking of actual system quality.

Define how zero-IDCG queries enter the aggregate

When every grade for a query is zero, both DCG and IDCG are zero, and direct division would be undefined. The policy here assigns that query score zero and retains it in the aggregate. It is an explicit evaluation convention rather than a unique consequence of the formula.

One ordinary query scoring one and one zero-IDCG query scoring zero produce macro-average 0.5 under this policy. Excluding the second query changes the average to one. The system output is identical, but the aggregate doubles. Report the total query count, zero-IDCG count and exclusion rule instead of choosing the larger value without disclosing the population change.

A separate issue is averaging per-query DCG/IDCG versus summing all numerators and denominators before division. For query scores 3/3 and 0/1, the equal-query macro-average is 0.5, whereas the pooled ratio is 3/4=0.75. Pooling effectively reweights queries by IDCG. It cannot be presented as the same equal-weight macro-average.

Nineteen checks make the evaluation contract executable

The program runs only Python standard-library arithmetic on an already ordered ID list. It loads no model and performs no vector search. It checks the hand-calculated numerator and denominator, ideal ordering, missing positions, zero gain, the candidate-only counterexample, order reversal, averaging differences and invalid inputs.

# Linear-gain, explicit-ranking teaching evaluator; not a search-engine run.
from math import log2,fsum,isclose
checks=0
def check(ok):
    global checks
    assert ok
    checks+=1
def dcg(grades,k):
    return fsum(g/log2(i+2) for i,g in enumerate(grades[:k]))
def ndcg(ranking,truth,k):
    if type(k) is not int or k<1:
        raise ValueError('k must be a positive integer')
    if any(type(g) is not int or not 0<=g<=3 for g in truth.values()):
        raise ValueError('grades must be integers in 0..3')
    if len(set(ranking))!=len(ranking):
        raise ValueError('duplicate document IDs')
    if any(d not in truth for d in ranking):
        raise ValueError('unjudged document: policy required')
    actual=dcg([truth[d] for d in ranking],k)
    ideal=dcg(sorted(truth.values(),reverse=True),k)
    return (actual/ideal if ideal else 0.),actual,ideal
truth={'A':3,'B':2,'C':1,'D':0}
s,actual,ideal=ndcg(['B','C'],truth,2)
check(isclose(actual,2+1/log2(3)))
check(isclose(ideal,3+2/log2(3)))
check(isclose(s,actual/ideal) and 0<s<1)
check(ndcg(['A','B'],truth,2)[0]==1.)
check(isclose(ndcg(['B'],truth,1)[0],2/3))
check(ndcg([],truth,2)[0]==0.)
check(ndcg(['D'],truth,2)[0]==0.)
check(ndcg(['B','C'],{'B':2,'C':1},2)[0]==1.)
check(ndcg(['C','B'],truth,2)[0]<s)
zero=ndcg(['Z'],{'Z':0},1)[0]
check(zero==0.)
check((1.+zero)/2==.5)  # Includes zero-IDCG query under this policy.
a=ndcg(['A'],{'A':3},1);b=ndcg([],{'B':1},1)
check((a[0]+b[0])/2==.5)
check((a[1]+b[1])/(a[2]+b[2])==.75)
for ranking,labels,k in [(['A','A'],truth,2),(['X'],truth,2),(['A'],truth,0),(['A'],truth,True),(['A'],{'A':-1},1),(['A'],{'A':1.5},1)]:
    try:ndcg(ranking,labels,k)
    except ValueError:check(True)
    else:check(False)
print(checks,'ranking-contract checks passed')
print('DCG',actual,'IDCG',ideal,'full-pool NDCG',s)
print('candidate-only NDCG',ndcg(['B','C'],{'B':2,'C':1},2)[0])

This small explicit-ranking evaluator differs from a library interface that accepts complete y_true and y_score matrices. If model scores tie, apply the project’s policy before producing the ordered list. The program does not compute an expectation over every tied permutation. The earlier top-k ties and recall discussion examines stable ordering and tie handling. Ignoring ties is not automatically a harmless performance optimization.

When changing the cutoff, judgment universe or deduplication policy, retaining these counterexamples makes definition changes easier to diagnose than inspecting only a large real-query average. The evaluator does not establish that a real RAG system achieved 0.617 or one. Every numerical result shown comes from deliberately constructed labels and rankings.

Ranking relevance is separate from answer evidence

Placing relevant documents early matters to retrieval, but NDCG does not inspect whether a generator cites the right sentence, remains faithful to the document or preserves important limitations. Observe retrieval relevance and answer evidence separately when evaluating a complete RAG pipeline. An offline ranking score is not answer accuracy.

Expanding a candidate window may change an evaluated ranking, but four documents establish no real latency or quality improvement. A comparison needs fixed queries, judgments, K, gain function, judgment pool and aggregation before reporting measured differences. The earlier Cross-Encoder reranking article covers the candidate-ordering pipeline; this article checks the scoring definition used to assess it.

If the denominator counterexample and reproducible code help identify an inflated retrieval score, voluntary support through the section below is welcome. The reusable pieces are a fixed judged universe, an explicit empty-query policy and retained query-level scoring records.

Sources and backfill note

NDCG, scikit-learn 1.7.2 · DCG, scikit-learn 1.7.2

Checked against versioned scikit-learn 1.7.2 definitions available before the intended date. Linear gains and explicit ID rankings are used here. Standard-library implementation, data and diagrams are original; no scikit-learn call or retrieval system ran. Article date: October 22, 2026 at 14:00 Beijing; actual backfill time is recorded separately and does not imply earlier publication.