Recall@k, MRR and nDCG

Recall@k does not measure how much of the relevant material you found. It measures how much of a list somebody wrote down you found. In trec_eval — the program that defines these metrics for the whole field — recall is rel_so_far / num_rel, and num_rel is the number of documents a human labelled, not the number that are relevant. Everything confusing about retrieval evaluation starts there.

That would be a footnote if the label sets were nearly complete. They are not. MS MARCO, the collection almost every retrieval model is trained and ranked on, has exactly one labelled passage for 94% of its development queries, out of 8.8 million passages. A chunk that answers the question perfectly and was never shown to an annotator is scored as a mistake, forever, by every system that finds it.

Underneath that there is a second problem, and it is the one that bites RAG specifically. These metrics score a ranking. A generator does not read a ranking. It reads the k chunks you paste into the prompt, all at once, as a pile of facts — and it can only answer if the facts it needs are somewhere in the pile. A ranking metric has no way to express "these two chunks together answer the question and either one alone does not."

Below is a 22-chunk API documentation corpus and six questions. Each chunk states a set of facts; each question needs a set of facts. The panel computes recall@k, MRR@10 and nDCG@k exactly as trec_eval does, alongside the thing you actually care about: whether the retrieved chunks contain everything needed to answer. Move relevance versus novelty from 0.50 to 1.00 and watch the two disagree.

At the defaults the panel reports nDCG@3 of 0.719, recall@3 of 0.365, MRR@10 of 1.000 — and all six questions are answerable. Now drag λ to 1.00, which is plain relevance ranking, the thing every vector database does by default. nDCG@3 goes to 1.000. Recall@3 goes to 0.673. MRR stays at 1.000. Every metric on the panel got better or held.

Three of the six questions can no longer be answered.

Look at the first row. The question needs two facts: the rate limit, and whether it is counted per key or per IP address. Five chunks in the corpus state the rate limit; two state the per-key rule. At λ = 1 the top three are quickstart#limits, help#rate-limit and onboarding#limits — three chunks that say the same number. All three are genuinely relevant, so recall@3 rises to 0.429 and nDCG@3 reaches a perfect 1.000, because the ideal top-3 is also three relevant documents and this ranking is one. The prompt now contains "100 requests per minute" three times and no answer to half the question.

Set λ back to 0.70 and compare carefully. Recall@3 is 0.673, MRR@10 is 1.000 and nDCG@3 is 1.000 — identical to λ = 1.00 on all three. Every question is answerable. Two rankings, indistinguishable to every metric in standard use, and one of them answers twice as many questions.

What the denominator actually is

Switch how the labels were made to "one answer per query" and read the label grid. Every row now has one green cell and as many as six amber ones. The amber chunks are relevant — they state facts the question needs — and the metric cannot see them. Then drag λ across its whole range. recall@3, MRR@10 and nDCG@3 sit at 1.000 for every value from 0.10 upward while the answerable count falls from six to three. The metrics are a flat line through a 50% collapse in the product.

That is not a contrived setting. It is the shape of MS MARCO. Arabzadeh, Vtyurina, Yan and Clarke measured it: "94% of the nearly seven thousand queries in the MS MARCO passage ranking development set have only a single known relevant passage, and no query has more than four." The passages were produced by extracting ten candidates from Bing results and asking an assessor to pick one containing an answer. Assessors were not asked to find every passage that answers, and not asked to find the best one.

So they went and checked. They put a modern neural ranker's top passage side by side with the official label and asked crowd workers which better answered the question. On the queries where the two differed, workers preferred the unlabelled passage 59% of the time — 43% against 30%, with the rest ties. Their conclusion is the title of the section: a real ranker can be "better than perfect", because a hypothetical ranker scoring MRR of 1.0 by always returning the official label would be returning the worse passage. Three runs near the top of the leaderboard significantly outperformed that perfect ranking under preference judgements. The RocketQA authors independently found that 70% of unlabelled top-ranked passages "are actually positives or highly relevant."

Two more clauses in trec_eval's source are worth knowing because they change what your averages mean.

Queries with no labels disappear. m_recall.c opens with if (0 == res_rels.num_rel) return (0) — the query contributes nothing and is not counted. If your evaluation set has queries whose answer nobody labelled, your reported recall is the recall over the easy subset, and adding hard queries to the set will not lower it.

Recall can be capped below 1.0 by arithmetic. BEIR ran into this and documented it: "consider a hypothetical dataset with 500 relevant documents for a query. Retrieving all relevant documents would produce a maximum R@100 score = 0.2, which is quite low and unintuitive." Their fix is capped recall, dividing by min(k, |relevant|). If you report recall@10 on a corpus where a typical question has thirty relevant chunks, your ceiling is 0.33 and a number like 0.31 is excellent, not terrible. BEIR's own table shows average relevant documents per query ranging from 1.0 on ArguAna to 38.2 on DBPedia — the same metric name means different things across those two rows.

Now switch the label policy to "pooled". The labels come from judging the top 5 of one baseline system, which is how TREC-style collections are built and how most teams build their first internal set: run what you have, label what it returns. Rows one, three and four drop from seven labels to five, because the baseline's top 5 was full of chunks stating the same fact and never surfaced the per-key rule at all. A different system that finds that chunk gets no credit for it. Pooling does not merely undercount; it undercounts in the direction of the system that built the pool.

What MRR can and cannot see

trec_eval documents reciprocal rank in its own help string: "Reciprocal Rank of the first relevant retrieved doc. Measure is most useful for tasks in which there is only one relevant doc, or the user only wants one relevant doc." That is an honest statement of scope, and it is the opposite of the RAG case, where you deliberately retrieve several chunks because you expect to need several.

Watch MRR in the panel while you move λ. From 0.10 to 1.00 it does not move at all. MMR's first pick is always the highest-relevance chunk regardless of λ — at the first step there is nothing selected yet, so the redundancy term is zero — and the first chunk is always labelled relevant. MRR is measuring a decision that none of the settings change. Ranks 2 through k, which is where the entire difference between an answerable and an unanswerable prompt lives, contribute nothing to it by construction.

MRR is the right metric for a navigational query: one correct answer exists, the user will click one result, and how far down they had to look is the cost. It is the wrong metric for a question whose answer is assembled from several documents, which is most questions worth building a RAG system for, and it is especially wrong for questions that need evidence from across a whole corpus.

nDCG, and which nDCG

nDCG — normalised discounted cumulative gain — is the metric BEIR chose as its single headline number, for a stated reason worth repeating: "Decision support metrics such as Precision and Recall which are both rank unaware are not suitable. Binary rank-aware metrics such as MRR and MAP fail to evaluate tasks with graded relevance judgements." nDCG is rank-aware and handles grades. It is the best of the three. It is also the one most likely to read 1.000 while your product is broken.

The discount is 1 / log2(rank + 1), straight out of m_ndcg.c, where the accumulation is results_gain / log2(i + 2) with i counting from zero. Rank 1 is worth 1.000, rank 2 is 0.631, rank 3 is 0.500, rank 5 is 0.387 and rank 10 is 0.289. Then the whole thing is divided by the same sum computed over the best possible ranking of the labelled documents.

That normalisation is where nDCG stops measuring what you think. If a question has seven labelled relevant chunks and you are computing nDCG@3, the ideal top-3 is three relevant chunks — so any three relevant chunks score 1.000. Three copies of the same fact score 1.000. This is the answer to "why is NDCG high even for wrongly ranked predictions", which is sitting unanswered on Data Science Stack Exchange: with binary or near-binary judgements and more relevant documents than your cutoff, nDCG@k saturates and stops discriminating. Drag k from 1 to 8 with λ at 1.00 and watch nDCG@k read 1.000 the entire way while the answerable count climbs from two questions to six.

And there are two nDCGs. trec_eval's gain is the relevance level itself — its help text says "Gain values are set to the appropriate relevance level by default" — so a grade-2 document is worth twice a grade-1. The learning-to-rank tradition uses 2rel − 1 instead, making grade 2 worth three times grade 1. Tick the exponential box and the panel's nDCG@3 moves from 0.719 to 0.733 on the same ranking, the same labels and the same cutoff. Fourteen thousandths is not much here because this corpus has few grade-2 chunks; on a collection with a real graded scale the gap is large. The point is not the size of the gap. The point is that "nDCG@10 = 0.412" is not a comparable number unless you also know the gain function, the cutoff, how ties were broken and how the ideal ranking was constructed.

Why the metric and the answer disagree

Collect the mechanism in one place. Retrieval metrics were designed for a user scanning a result list, where a second document saying the same thing as the first is a mild redundancy and the cost of a miss is one more scroll. A generator is not scanning. It gets k chunks in one context window and either the facts are there or they are not. So:

• Redundancy is free credit under every ranking metric and pure cost in a prompt. A near-duplicate of a relevant chunk is relevant, so it raises recall@k, holds nDCG at its ceiling, and consumes one of your k slots along with its share of the context you have to prefill.
• The metrics are computed per query and then averaged, which hides the shape. At λ = 1 the panel's mean nDCG is 1.000 and three individual questions are unanswerable. Any single number over a query set can move up while a subpopulation collapses; the per-query rows are the only place that is visible.
• Chunking changes the metric without changing the retriever. If the rate limit and the per-key rule had landed in one chunk instead of two, that question would be answerable at k = 1 under every setting here. Half of what a retrieval metric reports is a fact about where you cut the documents.

None of this means the metrics are useless. Two of them have a job that nothing else does. Recall at your candidate depth is a hard ceiling on everything downstream — it is the number that tells you whether a reranker can possibly help, because a reranker only reorders what recall already delivered. And comparing recall for two retrievers separately, before fusion, is the only way to find the queries where fusing two lists is worse than the better of its inputs. Use them as ceilings and as diagnostics. Do not use them as the thing you optimise.

Building an evaluation you can act on

Label by fact, not by document. For each question write down the facts required to answer it, then mark which chunks state each fact. This is more work than picking a relevant document and it is the only labelling that survives a chunking change, a reranker, or a duplicate page. Your headline metric becomes coverage: the fraction of required facts present anywhere in the top k. It is the hero readout in the panel and it is the one that tracks whether the generator can do its job.

Fifty queries, chosen adversarially, beats five hundred sampled. Deliberately include: questions whose answer is split across two documents; questions with an exact identifier in them; questions where the corpus contains a near-duplicate page and a canonical page; questions the corpus genuinely cannot answer, so you can measure whether the system says so. A uniform sample of production traffic is mostly easy queries and will report a high number that does not move when you change anything.

Report the distribution, never only the mean. Publish the count of queries at coverage 0, the count at partial coverage, and the count at full coverage, as three numbers that must each be watched. A change that moves ten queries from partial to full and three from full to zero is a regression, and it is invisible in any average.

Deduplicate before you evaluate, and again before you prompt. Near identical chunks are the single largest source of the gap this lesson is about. Cluster your chunks by embedding similarity above about 0.95, look at what falls out, and either merge them in the index or drop duplicates from the retrieved set before it reaches the model. MMR with λ around 0.7 is the cheap version of the same idea; the panel shows it costing nothing on any metric and buying three answerable questions.

Record recall at several depths, not one. Report it at your rerank depth, at your retrieval depth, and at a depth far beyond both. The gap between the first two is what reranking can win. The gap between the second and third is what widening retrieval can win. And the residual at the largest depth is what only fixing the index, the chunking or the query can win — some of which is candidates the approximate index dropped before anything else ran.

Check your labels the way the shallow-pooling authors did. Once a quarter, take the queries where your system's top result is not the labelled one, put the two side by side, and have a human pick. If your system wins more than it loses, your labels are stale and every number you have reported since the last check is understated. That is a cheap experiment and almost nobody runs it.