The Rewrite Replaces Your Question

In a conversational retrieval-augmented generation system — RAG, where retrieved text is put into a model's prompt before it answers — the retriever does not receive the user's question after the first turn. It receives one sentence written by a separate model call, and the user's own words are discarded before any search runs. There is no fallback. If that sentence is wrong, nothing downstream can recover, because nothing downstream ever saw what was asked.

Here is the whole of LangChain's create_history_aware_retriever, which is what almost every conversational RAG tutorial builds on:

retrieve_documents = RunnableBranch(
    (
        # Both empty string and empty list evaluate to False
        lambda x: not x.get("chat_history", False),
        # If no chat history, then we just pass input to retriever
        (lambda x: x["input"]) | retriever,
    ),
    # If chat history, then we pass inputs to LLM chain, then to retriever
    prompt | llm | StrOutputParser() | retriever,
)

Read the second branch. x["input"] — the thing the user typed — appears in the first branch and nowhere else. Once chat_history is non-empty, the retriever's entire input is StrOutputParser()'s output: a string a language model produced from a prompt you supplied. That model has never seen your index. It does not know which words your documents are written in. It is guessing at a search query for a corpus it has never read.

The prompt is where you might expect the corpus to enter, and it does not. create_history_aware_retriever takes whatever prompt you pass it, and its own docstring example pulls one off a hub (langchain-ai/chat-langchain-rephrase). LangChain's shipped default — CONDENSE_QUESTION_PROMPT, which ConversationalRetrievalChain.from_llm uses and which most tutorials copy verbatim — is six lines long, four of which are the labels and slots around the history, and its only instruction is "rephrase the follow up question to be a standalone question, in its original language". Nothing in it mentions retrieval, an index, or a vocabulary.

The older ConversationalRetrievalChain goes one step further. Its rephrase_question field defaults to True, and the docstring says what that means: "will pass the new generated question along" to the answering chain. So by default the model is not only searching for a question the user did not ask, it is answering one. The most-reacted bug report on the chain is titled "Why does ConversationalRetrievalChain rephrase every human question?", and its transcript shows the user typing Sure and the chain turning it into "Which of those activities is your personal favorite?"

Below is a five-turn conversation against an eleven-document corpus. The retriever is real Okapi BM25 — the ranking function that scores a document by how many of the query's rare words it contains, saturating on repetition and penalising length. Start with the history window at 0, which is the empty-chat_history branch: no rewriting, the user's words go straight through. Then move it to 1 and watch the hero number get worse.

At a window of 0, three of the five turns land the right document first. At a window of 1, two do. Rewriting made this conversation worse, and it did so while genuinely improving two of the five turns.

Both things are true at once, which is why this is hard to see in production. Turn 3 — "what about the non-interactive ones?" — is unanswerable raw: its document ranks 11th of 11 with a BM25 score of 0.00, because the question shares no term with it. At a window of 2 the rewriter lifts it to 2nd, at a score of 5.17. Turn 5 goes from 8th to 5th. Those are real gains. But turn 4 — "why does the gateway keep going after the client gives up?" — is a complete, self-contained question that needed no help at all, and the rewriter had no way to know that. It rewrote it anyway, because the branch condition is not x.get("chat_history", False) and nothing else. The result was "why does the session lifetime keep going after the client gives up?", and that phrase moved the wrong document to the top: the session-lifetime document scores 6.50 and the one that answers the question scores 3.04.

Now set the window to 2 and read the decision log for turn 4. The phrase it substituted, "the session lifetime", was not something the user said. It was introduced by the assistant at turn 3 — a turn whose retrieval had already gone wrong. That is the compounding: a bad retrieval produces an answer, the answer enters chat_history, and the next turn's rewriter treats it as the most salient thing in the conversation.

You can watch the cascade run forwards. Type service accounts into the override box on turn 3 — the correct search query, which we will come back to — and then look at turn 4's decision log again. The phrase it substitutes is now "the service account token", because that is what the corrected retrieval put into the assistant's mouth. Turn 4 is still wrong, and its document has moved from 2nd to 3rd. The problem there was never which topic got substituted. It was that anything did.

The rewriter speaks the conversation's language

Put your cursor in the override box on turn 3 and type the non-interactive ones — the user's actual phrase. The answering document ranks 11th of 11. Now replace it with service accounts. The same document ranks 1st, with a score of 6.08. Now try auth_token_ttl_service: also 1st.

Three phrasings of one question. Two of them are the corpus's own words and win outright; the third is the conversation's own words and loses to every document in the index. Nothing about the retriever changed between them.

This is the structural problem with putting a language model between the user and the index. The rewriter's job is to produce a query. Its inputs are the conversation and a prompt. Its inputs do not include the corpus, the vocabulary, the field names, the product's own nouns, or a single retrieved document. It resolves "the non-interactive ones" into whatever the conversation established that phrase to mean, which is the correct linguistic answer and the wrong search query — because your documentation says service account and your users say non-interactive, and the rewriter has only ever met the users.

That is also why the effect is worse on a BM25 leg than on a dense one. BM25 scores a term the query and the document share; share none and the score is exactly zero, with no partial credit. A dense retriever degrades more gracefully here, because "non-interactive" and "service account" are close in embedding space. It degrades. It does not stop degrading — and on the queries where the distinguishing term is an identifier rather than a concept, the dense leg has the same problem for its own reasons.

There is no window size that works

The obvious next move is to tune the history window. Send the rewriter fewer turns and it cannot drag old topics in; send it more and it can resolve references to things said earlier. Somewhere in between there should be a number.

Below, every window size is run against both conversations and every turn, and the cell shows where the answering document landed.

On the conversation whose subject changes, the best row is 0 — no rewriting — at three of five, and every window from 1 to 5 scores two. On the conversation that stays on one subject, window 0 scores three and every window from 1 to 5 scores four. The same rewriter, the same corpus, the same retriever, opposite conclusions.

The reason there is no interior optimum is that the window is being asked to do two incompatible jobs. Resolving a reference needs the window to reach back to wherever the antecedent was introduced, which argues for a long window. Not contaminating a new question needs the window to contain nothing from before the subject changed, which argues for a window of zero at exactly that turn. The right value is therefore a function of where the topic boundaries in the conversation are, and the rewriter is the component that would have to detect them — which is not what it was asked to do, and not something the branch condition not x.get("chat_history", False) can express.

Notice also which turns never turn green. Turn 3's document is at best 2nd at any window, because no amount of history supplies a word the conversation never contained. That failure is not a window-size problem and no window-size search will find it.

What the measurements actually say

None of this means rewriting is a mistake. The published numbers are unambiguous that it helps, and they are equally unambiguous about how much it still leaves on the table.

QReCC — a dataset of 14,000 conversations and 80,000 question-answer pairs over 54 million passages, built specifically to separate the rewriting subtask from the retrieval one — reports passage retrieval on the same queries three ways. On the raw conversational question, mean reciprocal rank (MRR, the average of one-over-the-rank of the first relevant passage) is 0.0343 and recall@100 is 11.71%. On the paper's best automatic rewrite: MRR 0.1586, recall@100 41.51%. On a human rewrite: MRR 0.1994, recall@100 49.36%. End to end, the F1 scores are 9.07, 19.10 and 21.82. The model there is the paper's own best rewriter, a 2020 encoder-decoder it calls Transformer++, so treat the model column as a floor rather than as what a current model would do.

Read those in the right order. Rewriting is worth roughly a 4.6× improvement in MRR over doing nothing, which is why everyone does it. And that rewriter still gave up another 26% of MRR and almost 8 points of recall@100 against a human doing the same task by hand, on a benchmark built by people who were trying to close exactly that gap.

The gap is not an artefact of one dataset or one rewriter. TREC CAsT, the conversational search track, measured the same shape independently across 80 topics of about ten turns each — 30 for training, 50 for evaluation. Its overview reports that "the results using the manually resolved queries demonstrates a gap of approximately 35% over the best automatic system", and separately that "the gap between the best manual and automatic runs is large, a 26% relative difference in median and 35% relative difference in the best runs". Two evaluations by different groups on different corpora, both finding a double-digit residual between a machine rewrite and a human one. Neither used a modern model, so the size of today's residual is an open question; its existence is not, because the thing that produces it — a rewriter that cannot see the corpus — has not changed.

So the honest statement is: rewriting converts an unusable retrieval into a mediocre one, and whatever residual your own rewriter leaves is systematic and invisible to everything downstream. Measuring it is step 4 below; no metric you are already collecting reports it. A cross-encoder reranker cannot recover it — the reranker reorders the candidates the rewritten query returned, so it inherits the rewrite whole. And HyDE does not address it either: HyDE changes what the query looks like to an encoder by generating a hypothetical answer to embed, which is a fix for the asymmetry between short questions and long passages. It has nothing to say about a pronoun, because it operates on the query it is given, and the query it is given is the one with the pronoun in it.

Checking it on your own system

All four of these are things you can do this afternoon, and the first one changes how the rest look.

1. Log the rewritten query. It is almost certainly not logged today. Set return_generated_question=True on ConversationalRetrievalChain, or attach a callback to the chat_retriever_chain run that create_history_aware_retriever names. Then put it next to the user's raw input in the same log line. Most conversational RAG deployments record the question and the answer and nothing in between, which means the one string that determined the whole turn is the one string nobody kept.
2. Diff the vocabularies. Take a week of rewritten queries, take your corpus, and list the query terms with zero document frequency. Those are the terms your rewriter is producing that your index cannot match, and they are usually a small, repeating set of user-side synonyms for things your documents call something else. That list is directly actionable: it is either a synonym map on the analyser or a paragraph of alternate phrasings added to the documents.
3. Measure per-turn, not per-conversation. Compute recall@k separately for turn 1 and turns 2 onward. Turn 1 goes through the no-history branch and its recall is your retriever's true capability; turns 2 onward go through the rewriter, and the difference between the two numbers is the rewriter's cost, isolated. A single averaged figure hides it completely, because turn 1 is usually the largest and easiest slice.
4. Run the ablation the benchmarks run. Take fifty real multi-turn questions, write the standalone version of each by hand, and retrieve with both. The gap between your model rewrite and your hand rewrite is your local version of QReCC's 0.1586-against-0.1994, and unlike the published number it tells you which of your turns are losing and why. If the gap is small, stop working on the rewriter. If it is large, the fix is usually not a better prompt — it is retrieving on the union of the raw question and the rewrite rather than on the rewrite alone, which costs one extra retrieval call and restores the fallback the RunnableBranch removed.

That last one is worth stating plainly, because it is the mitigation the framework's shape discourages. Nothing requires you to search on one string. Retrieve with the raw question and with the rewrite, fuse the two result lists, and a turn like turn 4 — self-contained and damaged by rewriting — survives, because the raw question is still in the pool. The branch in create_history_aware_retriever chooses between the two inputs. Choosing is the bug.