Dense Vector Retrieval

Cosine similarity does not measure whether a chunk answers your question. It measures the angle between two averages. Those are different things, and the gap between them is where most retrieval-augmented generation quietly fails.

A chunk becomes a vector by embedding every token and then pooling them — usually a plain mean — into one fixed-width vector, which is normalised to unit length so that cosine reduces to a dot product. That is the whole encoder as far as retrieval is concerned. Everything the ranking can possibly know about a 400-token passage has been averaged into a single direction, and the number you sort by is how closely that direction lines up with the average direction of your question.

Two consequences follow immediately, and neither is intuitive. A passage that restates your question points almost exactly where your question points, whether or not it answers it. And a passage that answers your question in one sentence out of forty has that sentence averaged down to a fortieth of its direction.

Below is a small corpus, embedded and ranked for real. Move the chunk size.

At the default 320-token chunks the top hit is faq §1 — a twelve-word index entry that restates the question and contains no answer at all. It scores 0.857 and it never moves, because it is twelve tokens long and stays twelve tokens long at every chunk size. The passage that actually answers the question sits below it and gets steadily worse as you widen the chunk, from 0.700 down to 0.622, without one word of it changing.

Now switch the query to a hypothetical answer and drag chunk size to its minimum. The answer reaches rank 1. Drag chunk size back up and watch it lose again. Nothing about the index, the model or the query changed on that second drag — only how much unrelated text was averaged in alongside the answer.

Why a restatement of the question wins

Pooling is an average, so a chunk's vector is the centre of mass of its tokens. The faq entry is almost the query itself: same words, same proportions, nothing else in the average to pull it anywhere. Its centre of mass lands on top of the query's centre of mass. That is the highest score the geometry can produce, and it is produced by a document that contains no information.

This is not an artefact of the toy corpus. Every documentation set is full of text that is shaped like a question: FAQ headings, section titles, issue titles, the "Symptoms" line of a runbook, the first sentence of a support ticket. Dense retrieval ranks all of them above the paragraph three screens down that resolves the problem, because relevance was never what was measured. Similarity was.

The practical form of this: compute the score of your own query against itself. That is the ceiling. Any chunk scoring close to it is close to being a paraphrase of your question, which is the strongest possible signal that it is not an answer. A retrieval system that treats 0.9 as "very relevant" has it precisely backwards at the top of the range.

Chunk size is a retrieval parameter, not an ingestion parameter

Chunk size is usually decided once, in the ingestion script, by someone thinking about the embedding model's token limit. The simulation shows what it is really controlling. Hold the query fixed on content words only and sweep chunk size: the answer's score falls from 0.512 to 0.291, a drop of more than 40%, purely from adding neighbouring text to the same chunk.

The mechanism is dilution. If the answer occupies a tokens of a chunk of n, the answer's contribution to the pooled vector is a/n of the sum, and the other (n−a)/n pulls the result toward whatever the neighbouring text is about. Because the vector is then normalised, this shows up as a rotation away from the query rather than as a smaller magnitude — the score drops without anything looking wrong.

How much it costs you depends on what the padding is about, which is exactly the thing you do not control when you split on a character count. In this corpus the answer's continuation drifts to a neighbouring section on certificates, so the dilution bites. Had the continuation stayed on topic, the same chunk size would have cost almost nothing. That variance is why "we tuned chunk size to 512" is not a finding that transfers between corpora, and it is the real subject of how documents get split.

Note also what the top_k slider does not fix. It changes how many chunks you forward, not their order. Raise it and the answer eventually gets into the context — behind a restatement of the question, and at a token cost the readout makes explicit. Those tokens are not free: every one of them is prefilled and held in the attention cache for the whole generation, and material placed in the middle of a long context is used less reliably than material at either end.

The rare token that carries all the information carries none of the vector

IL-4021 is the most informative thing in the query. It appears in two of the seven documents. Any human triaging this would search for that string and stop.

To the dense retriever it is one token among ten in the query and, at the default 320-token chunk size, one token among 320 in the chunk. Put the chunk slider back to 320 to read the log line for it — the last section walked it to the far end, where the same token is one in 600 and the log says 0.17% instead. It contributes about 0.3% of the pooled document vector. Worse, an identifier that the encoder never saw in training has no learned relationship to anything — it is subword fragments pointing in an essentially arbitrary direction — so its component in the query vector matches almost nothing in the corpus and does no discriminating work at all. It costs the answer chunk more than it gains it.

Turn on rank by BM25 instead, still at 320-token chunks. The ordering changes completely and the answer moves from rank 3 to rank 2. The log breaks its score down term by term, and the shape of that breakdown is the whole argument: IL-4021 occurs once in the chunk and is in two of the seven documents, and it contributes +1.02; worker and retry each occur twice and are in four of seven, and they contribute +0.72 apiece. Frequency in the chunk barely matters; rarity in the corpus decides. Now drag the chunk size down: the answer is first at every setting up to 280 tokens.

That advantage on rare tokens is why BM25 refuses to go away. The BEIR benchmark evaluated ten retrieval systems across eighteen datasets, and its abstract reports that "BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances" — which is to say the thing that reliably beats BM25 out of domain is a re-ranker, not a better bi-encoder.

Then drag back up. At 288 tokens BM25 loses the top spot to the faq restatement, and does not get it back at any larger size. Its length normalisation term — the b parameter, conventionally 0.75 — penalises the long answer chunk enough to lose to a twelve-token entry that matches three query terms. Neither retriever is a relevance model. Both are cheap proxies with different biases, which is the actual argument for running both and then rescoring the union.

Where the recall you measured is not the recall you get

Everything above assumes the index returned the true nearest neighbours. In production it usually does not. pgvector performs exact search with perfect recall by default, and its own README is explicit that adding an HNSW or IVFFlat index trades recall for speed and that you will see different results for the same query after adding one.

The failure that costs people the most days is the interaction with metadata filters. With an approximate index the filter is applied after the index scan. pgvector's README works the arithmetic: with the default hnsw.ef_search of 40, a condition matching 10% of rows leaves about four rows on average. Ask for top_k = 10 with a tenant filter and you get four chunks, no error, and a model that says it does not know. Raise hnsw.ef_search, or enable iterative index scans, and the same query starts working — which is why this gets misdiagnosed as an embedding-quality problem.

This is also the contrast with the other direction people take when ranking keeps failing: stop ranking. Read every community summary and merge the results, as in global search fan-out. That removes the ranking error and replaces it with a cost that scales with the corpus.

Checking it yourself

Four measurements, in the order that isolates the problem fastest:

• Log the scores, not just the chunks. If your top five all sit within about 0.03 of each other, the ranking is noise and the retriever is not choosing — it is reporting a tie. A reranker will change the answer; raising top_k will not.
• Embed the query and score it against itself, and against the top hit. A top hit close to the self-score is a paraphrase of the question. That single number distinguishes "found the answer" from "found the question" and takes one line to add.
• Re-run with the index dropped, or with hnsw.ef_search set high enough to force an effectively exact scan, and compare the returned ids. If they differ, your problem is approximate-search recall, not relevance. Do this before touching the embedding model.
• Grep for the rare token. If the string is in the corpus and the retriever did not return the chunk containing it, no amount of chunk-size tuning will fix it and you need a lexical index.

The prompt-side check is separate: where in the context a chunk landed changes whether the model uses it, so a retrieval fix and a context-ordering fix can look identical from the outside. Fix the ranking first, because you can measure it without the generator in the loop.

Next: hybrid retrieval and cross-encoder reranking — what reciprocal rank fusion actually computes, and why a reranker changes the answer when top_k does not. Then rewriting the query, which moves the one vector you control.