RAG / vector store / indexing
The Dimension Limit That Stops You Is Not the Column's
That an embedding model's dimension count is a quality dial, and that a vector database either accepts a dimension or rejects it. Both halves mislead. In pgvector the storage limit and the index limit are different constants in different headers: `VECTOR_MAX_DIM 16000` in `src/vector.h` bounds the column, while `HNSW_MAX_DIM 2000` in `src/hnsw.h` and `IVFFLAT_MAX_DIM 2000` in `src/ivfflat.h` bound the index — and `hnswbuild.c` raises `column cannot have more than %d dimensions for hnsw index` only when you run CREATE INDEX, which for many teams is after the data is loaded and the feature has shipped. A 3,072-dimension column therefore works perfectly at small scale and degrades into an exhaustive scan of every row as the table grows, with no error anywhere. The second half is that the number itself is not the quality knob people think: the HNSW paper is explicit that what matters is a dataset's effective dimensionality across scales, not its column width, so whether truncating 3,072 to 1,024 costs you recall is a measurement on your corpus and your model, and cannot be read off a number.
pgvector has two unrelated dimension ceilings — VECTOR_MAX_DIM of 16000 on what a column may store, and a per-index-type, per-vector-type maxDimensions of 2000 for vector, 4000 for halfvec and 64000 for bit — so a 3,072-dimension embedding inserts without complaint and cannot be given an approximate index at all, which surfaces as a sequential scan in production rather than as an error at write time.
The lesson that sent you here
- chunk_size Is Measured in the Wrong Units
That chunk_size controls how much text ends up in a chunk's vector. It controls how much text ends up in the chunk; whether the encoder reads all of it is a separate limit measured in a different unit. all-MiniLM-L6-v2 stops at 256 word pieces — 254 once [CLS] and [SEP] are counted — and its model card describes this in one sentence with no warning, no exception and no truncated flag. The lost text is still in your vector store, still returned verbatim once the chunk is retrieved, so it reads correctly in every debugging session; it simply had no influence on the vector that decides whether the chunk is ever retrieved. And the token-aware fix fails: setting chunk_size to 256 in LlamaIndex's cl100k tokens produces chunks of 278, 292 and 313 word pieces on three real documents, all of them over the 254 the encoder will read.
1 published lesson depends on this concept, which is what moves it up the writing queue. Nothing is hidden behind this page — it has not been written.