chunk_size Is Measured in the Wrong Units

You set chunk_size = 1000. Three different pieces of software now hold three different opinions about what that number counts, and the one that decides whether your text ends up in a vector is the one you never configured. On a document of log lines, a 1,000-character chunk arrives at the encoder as 456 word pieces, the encoder reads the first 254, and the remaining 580 characters are discarded without an error, a warning, or a field in the response.

The text is not lost. It is still in the vector store, still returned verbatim the moment that chunk is retrieved, still visible in every debugging session you will run. It simply had no influence on the vector that decides whether the chunk is ever retrieved at all.

This is a different failure from cutting an answer across a chunk boundary. There the answer is split between two chunks and neither contains it. Here one chunk contains the whole answer, prints the whole answer, and is not findable, because the half of it holding the answer was never read by the model that produced its vector.

Three rulers

A chunk passes through three measurements on its way into an index, and each one is taken with a different instrument.

The splitter measures first. LangChain's TextSplitter base class — the one RecursiveCharacterTextSplitter inherits from — declares its constructor as chunk_size: int = 4000, chunk_overlap: int = 200, length_function: Callable[[str], int] = len. The default length_function is Python's len, so chunk_size counts characters, and the docstring for the next argument says so out loud: "Overlap in characters between chunks." LlamaIndex's SentenceSplitter makes the opposite choice. Its chunk_size field is documented as "The token chunk size for each chunk", it defaults to DEFAULT_CHUNK_SIZE = 1024 # tokens, and the tokens it counts come from tiktoken under encoding_for_model("gpt-3.5-turbo"), which resolves to the cl100k_base encoding and its 100,277-entry vocabulary.

The encoder measures second, and it is the only measurement that binds. A sentence-transformers model carries a file called sentence_bert_config.json. For all-MiniLM-L6-v2 that file contains exactly two keys, one of which is "max_seq_length": 256. The model card states the consequence in a single sentence: "By default, input text longer than 256 word pieces is truncated." Not rejected. Truncated. The sentence-transformers documentation says the same thing from the other side — "Longer texts will be truncated to the first model.max_seq_length tokens."

Those word pieces are not cl100k tokens. They come from BertTokenizer running WordPiece over a 30,522-entry vocabulary, a third of the size of cl100k_base's. And the budget is 254, not 256, because [CLS] and [SEP] are prepended and appended and count against the limit.

Below is a real splitter feeding a real encoder budget. The three documents are written for this page, but every token count in the simulation was produced offline by tiktoken's cl100k_base and by BERT's own WordPiece algorithm over all-MiniLM-L6-v2's shipped 30,522-piece vocabulary, then shipped here as a per-piece offset table. Start with the defaults — runbook prose, 1,000 characters — and read the top-left readout. Everything is fine. Then drag chunk_size to 1,500.

At 1,000 characters of runbook prose, the largest chunk is 206 word pieces against a budget of 254 and nothing is lost. At 1,500 the largest is 305, both chunks are over, 342 characters — 12% of the document — are in the store and not in any vector, and the answer to the question is one of them. Nothing about that transition is announced. The ingestion job's log line still reads two chunks written.

Now change the document to the incident log and leave chunk_size at 1,000. Same splitter, same number, same encoder. The largest chunk is 456 word pieces, and 580 characters — 28% of the document — never reach the encoder. The reason is in the last readout: this document spends 2.85 characters per word piece where the prose spent 4.95. A UUID like 8f3c9a21-4bd7-4f1e-9c30-2ab77e5d1108 is one glance to a human and 32 word pieces to the tokenizer. Push it to 2,000 characters and 75% of the incident log is outside its own vectors.

That ratio is not a property of your splitter or of your embedding model. It is a property of your text, and it is the reason chunk_size = 1000 can be correct in the documentation corpus and catastrophic in the log corpus sitting in the same index.

The token-aware fix does not fix it

The obvious response is to stop counting characters. Use a splitter that counts tokens, set the count to the encoder's limit, and the mismatch disappears. It does not.

Switch the unit to cl100k tokens and set chunk_size to exactly 256, the number max_seq_length advertises. Then read the largest-chunk figure on each of the three documents. Runbook prose: 278 word pieces. Incident log: 292. Config reference: 313. The budget is 254. All three overflow, and the config reference overflows it by 23%.

The reason is that cl100k_base and WordPiece are different algorithms over different vocabularies, and neither is a scaled version of the other. cl100k_base is byte-pair encoding over 100,277 entries. all-MiniLM-L6-v2 tokenizes with WordPiece over 30,522. A vocabulary a third of the size has to spell more words out of fragments, so it emits more pieces for the same string — but how many more depends entirely on which strings.

The word certificate is one token in both. The word Kubernetes is two cl100k tokens (K + ubernetes) and four word pieces (ku + ##ber + ##net + ##es). The configuration key autovacuum_vacuum_scale_factor is nine cl100k tokens and ten word pieces. Aggregate that over a whole document and the ratio lands wherever the document's vocabulary puts it: 1.07 pieces per token on the runbook prose, 1.10 on the incident log, and 1.19 on the config reference. There is no constant to multiply by.

So a token-counting splitter set to the encoder's own advertised limit still overflows it, by an amount your corpus decides. The splitter is counting honestly. It is counting the wrong thing honestly.

The number you want does not exist as a number

Below, the same machinery sweeps chunk_size across its whole range for all three documents at once and marks, for each size, how much of that document ends up outside its own vectors. There is a largest safe size for each document. There is no largest safe size for the index.

With all-MiniLM-L6-v2 and a character-counting splitter, the largest size that truncates nothing is 1,240 characters for the runbook prose, 950 for the config reference and 420 for the incident log — a spread of 2.95× across three documents that would sit in the same collection. Switch to all-mpnet-base-v2, whose limit is 50% higher, and the safe sizes become 1,870, 1,520 and 690: every one of them larger, and the spread still 2.71×. A bigger encoder budget buys headroom. It does not buy a single number that is right for the whole corpus, because it does not change the thing that varies.

Switch the unit to cl100k tokens and the safe sizes are 236, 205 and 226. They are closer together, which is the real and modest benefit of a token-aware splitter, and all three are still below the 254 the encoder will actually read.

Two failure modes, and you have only ever seen one

An engineer whose entire RAG — retrieval-augmented generation, where retrieved text is placed in a model's prompt before it answers — experience is on a hosted embedding API has never seen this failure, because hosted APIs do not truncate. They refuse. OpenAI's own cookbook sets EMBEDDING_CTX_LENGTH = 8191 for text-embedding-3-small, sends an over-length string, and commits the response into the notebook:

Error code: 400 - {'error': {'message': "This model's maximum context
length is 8192 tokens, however you requested 10001 tokens (10001 in your
prompt; 0 for the completion). Please reduce your prompt; or completion
length.", 'type': 'invalid_request_error', ...}}

That error is a gift. It stops the ingestion job, names the limit, names the overage, and cannot be ignored. Select the OpenAI encoder in either simulation above and the loss goes to zero at every chunk size, because 8,191 tokens is about 43,600 characters at this prose’s measured rate of 5.32 characters per cl100k token, and the largest chunk anything on this page produces is 671. The limit exists; you will simply never reach it with anything a splitter produced.

Then the team moves to a self-hosted encoder to cut cost or to keep text inside the network. all-MiniLM-L6-v2 is the obvious first choice: 384 dimensions, small enough to run on a CPU, and the model every tutorial reaches for. Its budget is not 8,191 tokens. It is 254 word pieces, and a word piece covers fewer characters than a cl100k token does, so the drop is larger than the ratio of the two numbers. In characters, on the runbook prose above: 1,258 against 43,574, which is 35 times less text — and 38 times less on the config reference, whose word pieces are shorter still. The failure it produces is not an error but a silence. The job completes. The chunk count is right. The text reads back correctly. Retrieval quality drops and the ticket that gets filed says the new embedding model is worse.

It is worth being precise about what "worse" means here, because the diagnosis is often mis-assigned to the encoder's notion of similarity. The encoder is behaving exactly as documented. The chunk that lost its tail is not scored badly; it is scored accurately, as a representation of the text the encoder was given, which is the first 254 pieces. Everything downstream inherits that. A cross-encoder reranker cannot fix it, because the reranker only reorders a candidate list the truncated vector failed to get into. Fusing in BM25 can genuinely help, and it is the one mitigation on this list that does, precisely because BM25 indexes the literal tokens of the whole chunk and has no sequence limit at all — but it hides the bug rather than removing it, and it only helps the queries that share a rare literal term with the lost text.

What the loss is not

Two honest caveats, because this topic attracts more folklore than any other in retrieval and the folklore is mostly claims nobody measured.

Truncation is not the same as a bad chunking strategy, and fixing it is not the same as adopting one. Semantic chunking — splitting where the embedding similarity between consecutive sentences dips, rather than at a fixed size — is the most-recommended remedy in the blog literature. When it was evaluated properly across document retrieval, evidence retrieval and answer generation, the authors of "Is Semantic Chunking Worth the Computational Cost?" concluded that "the computational costs associated with semantic chunking are not justified by consistent performance gains". It costs an embedding pass over every sentence in your corpus. It does not reliably beat a fixed window. It also does nothing at all about the problem on this page, because a semantic boundary is still a boundary measured in characters, and the encoder's limit is still counted in word pieces.

Not all lost text costs you a retrieval. The tail of a chunk is often boilerplate, and a document whose last 12% is a footer loses nothing by dropping it. The measurement that matters is not the truncated percentage. It is whether the spans that answer real questions are on the wrong side of the cut — which is why the hero readout in the first simulation tracks one specific answer span rather than the loss figure, and why those two readouts routinely disagree. At 1,200 characters on the incident log, 37% of the document is outside its own vectors and the answer is still findable. At 1,500 characters on the runbook prose only 12% is lost, and the answer is one of the casualties. The percentage is a symptom; the span is the outcome.

Checking it on your own system

This takes about ten minutes and does not require an evaluation set.

1. Print the limit you are actually running under. print(model.max_seq_length) on a SentenceTransformer. Do not read it off the model card for the architecture — read it off the object, because the value comes from sentence_bert_config.json in the specific snapshot you downloaded, and it is frequently lower than the transformer supports. all-MiniLM-L6-v2's tokenizer config declares model_max_length: 512 while its sentence_bert_config.json declares max_seq_length: 256. The 256 wins. Subtract two for [CLS] and [SEP].
2. Measure your corpus, not a rule of thumb. Run len(model.tokenizer(text)["input_ids"]) over a few hundred real chunks and divide the character count by it. The widely repeated "about four characters per token" is a statement about English prose. The three documents on this page measure 4.95, 3.81 and 2.85 characters per word piece, and the rule of thumb is wrong about all three.
3. Count the overflow directly. This is the number that belongs on the same dashboard as recall@k, because no retrieval metric can see it — a chunk that was never fully embedded is simply absent from the results being scored. One line: sum(len(model.tokenizer(c)["input_ids"]) > model.max_seq_length for c in chunks). If that is not zero, you have a number for how many of your chunks are partially indexed. Nothing in your pipeline is producing this number for you today.
4. Find the questions it costs you. For each over-length chunk, truncate the text yourself with model.tokenizer.decode(model.tokenizer(c)["input_ids"][1:max_len-1]) and diff it against the chunk. That diff is the text your index cannot see. Grep it for the identifiers, error codes and version numbers your users search by; those are the rare literal tokens that carry the most retrieval signal per character, and they cluster in the tails of log and reference documents rather than in the openings.
5. Then set the size from the measurement. Safe chunk_size in characters is roughly (max_seq_length − 2) × chars_per_piece from step 2, computed per document type rather than once for the corpus. If two document types in one collection disagree by a factor of three — as the runbook and the log do here — that is a signal to split them into two ingestion configurations, not to average the two numbers.

One thing not to do: raising model.max_seq_length past what the checkpoint was trained on. The sentence-transformers documentation permits the assignment and warns about it in the same breath — "You cannot increase the length higher than what is maximally supported by the respective transformer model", and models trained on short texts do not produce good representations of long ones. Raising a MiniLM from 256 to 512 stops the truncation and gives you vectors for a length distribution the model never saw. Choose an encoder with the budget you need, or make the chunks fit the one you have.

Once the text is genuinely inside its own vector, the next thing that decides whether it is findable is how many numbers that vector gets to use — which is its own quiet trade — and then what averaging those numbers together does to a rare token that carried all the meaning. Truncation is the failure that comes before both, and it is the only one of the three that is a bug rather than a trade-off.