Late Interaction and MaxSim

ColBERT never reads your query and your document together. Its document encoder runs offline, on the document alone, with no idea what anyone will ever ask — exactly like the model that filled your vector index. The word "late" describes when the pooling happens, not when the reading happens, and almost everything surprising about late interaction follows from that one distinction.

A normal dense retriever squashes a chunk into a single vector before any query exists — that is one vector per chunk, the assumption every vector database is built on. ColBERT — Khattab and Zaharia's 2020 retriever, the name is a contraction of "contextualized late interaction over BERT", where BERT is Bidirectional Encoder Representations from Transformers, the language model it encodes with — keeps one vector per token instead, and defers the squashing until scoring time. It then scores a query-document pair with an operator called MaxSim: for each query vector, find the single document vector it matches best, and add those best matches up.

S_q,d = ∑_{i ∈ |Eq|} max_{j ∈ |Ed|} E_qi · E_dj

Read that formula for what is missing rather than what is present. There is a sum over query vectors and a maximum over document vectors. There is no term anywhere that depends on how long the document is. Hold that thought and open the panel below. It has the same eleven-chunk corpus scored four different ways; start with the scorer on one pooled vector per chunk — the mean-pooled cosine you already live with — and read the ranking.

With the pooled scorer, refund#policy — "Refunds are issued to the original payment method within five business days" — is rank 1 at 0.980. That is the right answer and it is comfortably first. Now switch the scorer to MaxSim and change nothing else.

It falls to rank 3, behind handbook#all, a 46-token everything-page, and refund#faq, which restates the question and links elsewhere. Directly beneath it at rank 4 is api#keys: 43 tokens about rotating API keys, with no connection to refunds whatsoever, which the pooled scorer put at rank 8.

Look at the token counts in the right-hand column of the ranking. The two longest chunks in the corpus, at 46 and 43 tokens, were ranks 5 and 8 under pooling. Under MaxSim they are ranks 1 and 4.

Nothing about the documents changed. Nothing about the query changed. The embeddings are identical — the same term vectors, the same corpus, the same arithmetic. All that changed is whether the vectors get averaged before the comparison or after it, and that alone reordered the list in favour of length. The rest of this lesson is about why that is not a bug in the implementation, why nobody has fixed it, and what you are supposed to do instead.

What "late" actually refers to

The confusion worth clearing first is that late interaction is a cheap cross-encoder. It is not, and the difference is not a matter of degree.

A cross-encoder concatenates the query and the document into one sequence and runs the whole thing through a transformer, so every query token can attend to every document token and back again. The document's internal representation therefore depends on the query. That is precisely why it cannot be precomputed: there is nothing to store until somebody asks a question. The attention pattern over the concatenated pair is the entire product.

ColBERT does no such thing. The paper's own sentence is that it "independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step". Independently. The document goes through BERT once, at index time, alone. Its vectors are frozen on disk before your query exists. No query vector has ever attended to a document vector and none ever will — the only thing that happens at query time is a matrix of dot products, a maximum down one axis and a sum down the other.

So the ordering is: a bi-encoder pools early and compares one pair of vectors; ColBERT pools late and compares |Eq| × |Ed| pairs of vectors; a cross-encoder never pools at all and computes a fresh representation per pair. Late interaction sits in the middle by giving up the joint forward pass, and it buys a great deal for that. On MS MARCO — Microsoft's passage-ranking benchmark, 8.8 million short passages — the paper measured BERT-base re-ranking the official top-1000 candidate list at 10,700 ms per query on a single Tesla V100, and ColBERT doing the same job at 61 ms. That is 175× on latency and, by Table 1's count, 13,900× fewer FLOPs — floating-point operations — per query.

The part that gets dropped when this is retold: it is not free on quality. Table 1 of the paper reports ColBERT at 34.9 MRR@10 — mean reciprocal rank at 10, the average of 1 divided by the position of the first correct passage, scoring zero if it is not in the top ten — against Nogueira and Cho's BERT-base at 34.7, which reads like a clean win. But the same table also carries "BERT_base (our training)" — the same cross-encoder architecture trained with ColBERT's own loss for a fair comparison — at 36.0. Against the matched baseline, late interaction gives up 1.1 MRR@10 points to go 175× faster. That is a very good trade and it is still a trade.

The four details that decide the numbers

• Query augmentation. The query is padded with BERT's [mask] tokens up to a fixed length, N_q = 32 in the paper, and the embeddings produced at those mask positions are kept and scored like any other query vector. The paper calls this "a soft, differentiable mechanism for learning to expand queries", and its ablation shows MRR@10 drops noticeably without it. A three-word query still produces 32 query vectors.
• A projection down to 128 dimensions. BERT's 768-dimensional output is passed through a linear layer to 128 dimensions. This barely affects query encoding cost; it exists almost entirely to control how big the index gets.
• L2 normalisation. Every output embedding is scaled to unit length, so a dot product is a cosine in [−1, 1]. The MaxSim of a 32-vector query is therefore bounded by 32 and floored by −32, and in practice sits somewhere in the teens or twenties.
• Punctuation filtering, documents only. The document encoder drops the embeddings for punctuation symbols to shrink the index. Documents are never padded with masks; only queries are.

That asymmetry is worth stating plainly because it is the source of a whole class of confusion: the query side has a fixed number of vectors and the document side does not. The sum in MaxSim runs over a constant. The maximum runs over a variable.

The missing denominator

Set the scorer to MaxSim, leave the query on the refund question, set append a see also footer to to the chunk that answers the question, and walk the footer slider from 0 to 5.

The answer's MaxSim goes 7.269, 7.323, 7.323, 7.361, 7.416, 7.480. It rises or holds at every step and never falls, and its rank improves from 3 to 2. Now switch the scorer to one pooled vector per chunk and walk the same slider: 0.980, 0.952, 0.928, 0.920, 0.914, 0.919, with the rank going 1, 2, 2, 4, 4, 4.

Same document. Same appended text. Same five sentences of navigation chrome that answer nothing. One scorer rewards it and the other punishes it, and the two curves run in opposite directions for the whole length of the slider.

Note also what the padded answer cannot do: even at five footer sentences it is still second, because handbook#all is longer and gets the same treatment. You cannot out-pad the everything-page. That is worth knowing before anyone proposes fixing relevance by making the good documents longer.

The pooled score fell because a mean gives every token weight 1/n. Twelve tokens about refunds average to a vector that points at refunds; add forty-five tokens about rate limits and support hours and documentation review and the mean drifts away from refunds, because that is what a mean does. This is the same mechanism as pooling dilution, seen from the ingestion side.

The MaxSim score rose, and here is the part that is not an empirical tendency but an identity. MaxSim takes, for each query vector, the maximum over the document's vectors. Appending text to a document does not modify the set of vectors it already had; it adds new ones. A maximum taken over a superset is greater than or equal to the maximum taken over the subset. Term by term, therefore:

D ⊆ D′  ⇒  max_{d ∈ D′} sim(q, d) ≥ max_{d ∈ D} sim(q, d)   for every q  ⇒  S_q,D′ ≥ S_q,D

Adding text to a document cannot lower its MaxSim score. Not for this query, not for any query, not by any amount. There is no configuration of ColBERT in which it can. The only escape is the one noted in the panel's caption: real ColBERT vectors are contextualized, so appending text also perturbs the existing tokens' vectors slightly, and the inequality becomes overwhelmingly likely rather than certain. The scoring function still has no denominator.

The practical form of this is worth saying in one sentence, because it is actionable and almost nobody is told it: under late interaction, the "see also" footer at the bottom of your documentation page is a ranking signal. So is the nav sidebar, if your extractor kept it. So is the boilerplate legal footer, the "was this helpful?" widget, and the auto-generated list of related articles. Under a pooled embedding all of that is a penalty and you may well have spent time stripping it. Under ColBERT you have been stripping your own ranking signal, and the everything-page you were embarrassed about is now your best-performing document.

Set the footer target to handbook#all and watch that directly. Its MaxSim goes 7.505, 7.505, 7.505, 7.523, 7.523, 7.553 — flat, flat, then up, never down. The flat steps are the inequality doing exactly what it says: those footer tokens failed to beat any existing maximum, so they contributed nothing, and "nothing" is the floor. And note where it started. At zero footer sentences the everything-page was already rank 1. It does not need padding to win; padding is only the proof that it cannot lose.

The two obvious fixes, and why neither ships

If the problem is a missing denominator, add one. Both natural ways of doing that are in the panel.

Divide by document length. Select MaxSim divided by chunk length. The everything-page collapses from rank 1 to rank 11 and api#keys from rank 4 to rank 10, which is what you wanted. Now look at what is rank 1: support#hours, "Support answers on business days between nine and five", nine tokens, nothing to do with refunds. It wins because dividing a sum of maxima by document length does not measure relevance density, it measures shortness. Try the other three queries — that same nine-token chunk is rank 1 for every single one of them, including the question about HTTP 429. You have not removed the length bias, you have inverted its sign.

Use an average instead of a maximum. Select sum of per-token averages: for each query vector, average its similarity against every document vector instead of taking the best one. This is length-normalised by construction, and it works on the symptom — handbook#all drops to rank 6, api#keys to rank 7, and the correct answer comes back to rank 2. But precision goes with it. Averaging means a document is rewarded for being uniformly vaguely related and gets no credit for containing the exact term you asked about; a single perfect match is diluted by every irrelevant token sitting next to it, which is the pooling problem reintroduced one query-token at a time.

This is not a hypothetical. The ColBERT paper ran exactly this ablation — model [B], "replaces ColBERT's maximum similarity with average similarity" — and reports that the results "suggest the importance of individual terms in the query paying special attention to particular terms in the document." Maximum beat average, and the design shipped with the maximum, length bias included. The same ablation table also answers the other obvious question: model [A] gives the document a single 4096-dimensional vector, which is N_q × 128 and so has the same parameter budget as 32 ColBERT vectors, and it is "considerably less effective". More dimensions in one vector does not substitute for several vectors. The granularity is the point.

So the length sensitivity is not an oversight anyone forgot to patch. It is the cost side of a trade that was measured, found favourable on MS MARCO, and shipped. Your corpus is not MS MARCO. MS MARCO passages are short and roughly uniform — the paper's own figures work out to about 68 embeddings per passage — so the length term that MaxSim omits barely varies there. A documentation corpus where a chunk can be 40 tokens or 900 is a different distribution entirely, and the omission that was harmless in the benchmark is the dominant effect in your index.

One vector per token is an index problem

The second thing people discover late is what this does to storage, and the arithmetic is worth doing once because it decides whether ColBERT is deployable for you at all.

A conventional index stores one vector per chunk. ColBERT stores one per token. Table 4 of the paper gives the MS MARCO figures directly: at 128 dimensions and 4 bytes per dimension — 512 bytes per token — the index is 286 GiB. Work backwards and that is 600 million embeddings across 8.8 million passages, about 68 per passage. Against a single-vector index of the same collection at 768 dimensions and 4 bytes each, which is 3,072 bytes per passage and 25.3 GiB in total, ColBERT is 11× larger.

Dropping to 2-byte floats gets it to 143 GiB. Dropping to 24 dimensions and 2 bytes gets it to 27 GiB at a cost of 1.0 MRR@10 point, which the paper notes with some satisfaction. ColBERTv2 does much better than either by encoding each vector as the index of its nearest centroid plus a quantized residual: 4 bytes for the centroid id and 16 or 32 bytes for the residual at 1 or 2 bits per dimension, so 20 or 36 bytes per vector against v1's 256 at 16-bit precision. That takes the MS MARCO index from 154 GiB to 16 or 25 GiB — the paper's headline 6–10× — including 4.5 GiB for the inverted list, the map from each centroid to the embedding ids assigned to it.

Even compressed, the shape of the data is wrong for most vector databases, and this is why the question "how do I store ColBERT embeddings" keeps getting asked and keeps not getting answered. A typical vector field is one row, one vector, one id. Late interaction needs one row to own a variable-length set of vectors, retrieval to happen at the vector level, and scores to be grouped back up to the row level with a max-reduce before they are summed. That is not a filter you can add to a k-nearest-neighbour query; it is a different execution model. Vespa, LintDB, VectorChord and a growing list of others exist because of it.

End-to-end retrieval — using ColBERT as the first stage rather than as a reranker — makes the same point in a different register. ColBERTv2 finds the nearest centroids for each query vector, walks the inverted list to the passage embeddings near them, decompresses those, and max-reduces by passage id. So approximate-index recall enters your pipeline once per query vector rather than once per query, and the paper is explicit that this computes a lower bound on the true MaxSim, since a document vector that no probe reached simply does not participate in its own maximum. Most teams skip all of this and run late interaction as a second stage over someone else's candidate list, in which case the candidate window bounds everything, exactly as it does for a cross-encoder.

Checking it on a real system

Four things to measure, in the order that finds bugs fastest.

1. Correlate rank with chunk length. Take a few hundred production queries, retrieve the top 10, and compute the rank correlation between a chunk's position and its token count. On a healthy index it is near zero. If it is strongly negative — long chunks reliably ranking high — you are reading the effect in this lesson and not a relevance signal. This single number is the fastest diagnostic there is and it needs no labels, because it is a property of the results rather than of their correctness.

2. Stop thresholding the score. Move the [MASK] vectors slider in the panel and watch every score in the list climb together: each extra query vector adds another term to a sum that has no normaliser. A real ColBERT pads to N_q = 32, so its scores routinely land in the teens or twenties, and the floor is large because generic query vectors always find something to match. A MaxSim of 18 means nothing on its own; it is not comparable to a MaxSim of 18 from a different query, a different N_q or a differently-chunked corpus. Every "minimum relevance score" cutoff you would write for cosine similarity is meaningless here. Use ranks, and measure them with recall@k against labelled queries.

3. Log the argmax, not just the score. The argmax is which document vector won each of those maxima, and it is the diagnostic late interaction gives you that no other retriever can. For a bad result, dump which document token each query vector matched — the panel's alignment columns are exactly this view. Two patterns are worth grepping for. If many query vectors collapse onto the same document token, the document is winning on one word and the sum is telling you a story about a single match. And if the winning matches are all near-misses in the 0.8s while a rival document has exact 1.0 matches on the terms that actually matter, you are watching a long document accumulate mediocre maxima and beat a short one with excellent ones. Switch the query to does the old api key still work after rotating it to see the legible version: api#keys matches the, old, api, key and after at exactly 1.000 and rotating to rotate at 0.986, which is what a correct alignment looks like.

4. Re-open the chunking decision. Much of the received advice about chunk size is really advice about pooling: cut small so the mean is not diluted. Late interaction deletes that argument and replaces it with the opposite one, so if you migrated an index without revisiting the splitter you are carrying a parameter tuned for a scorer you no longer use. The new constraint is uniformity rather than smallness — MaxSim compares documents on an axis it does not normalise, so what hurts you is variance in chunk length across the corpus, not the absolute value. Chunking to a consistent token budget matters more here than it ever did for a one-vector-per-chunk index, and stripping boilerplate stops being cosmetic.

When to reach for it at all. Late interaction is strongest where a query hinges on a term that must actually appear — a part number, an error code, a proper noun — because a pooled vector averages that term away and MaxSim compares it directly. That is the same weakness hybrid retrieval attacks by keeping a BM25 channel, and the two are worth benchmarking against each other before you take on an index eleven times the size; a lexical retriever costs far less and never paraphrases. It is worth knowing that the ColBERTv2 authors are candid about this. Their own paper observes that recent single-vector models with well-tuned supervision "sometimes perform on-par or even better than 'vanilla' late interaction models", and that these gains "challenge the value of fine-grained late interaction". The architecture is a real advance and it is not a free upgrade over a well-trained bi-encoder.