[jeadie]

I'm skeptical about some vector databases these days, but your article misses a few import points when it comes to LLMs.

1. To use LLMs effectively, you often need to generate and store more than 1 vector per document. 10 million vectors may only be 100,000 documents. This may still be enough for alot of small problems. 2. Pgvector currently has great limitations on recall/latency because underlying its ANN its using IVF (I'm currently working on adding HNSW-IVF and HNSW support to PGVector). In some cases, even elasticsearch can have issues with scale (the problem comes from the constraint of one ANN index per index segment, and immutability). 3. Pre-calculate seems like the wrong word to describe HNSW graph construction.

I think a point you miss that is important to consider for LLM + vector DBs is the fact that so much of the complexity of these uses cases cannot be captured by the vector DB (e.g. pinecone, chroma, qdrant, etc). I think there are some more end to end systems, at least in search, attempting to solve this (e.g Marqo, maybe Weaviate). Overall, I like the article. It makes a worthwhile claim and counterpoint to all the vector DB hype.

[kacperlukawski]

I'd love to hear more about your thoughts on the complexity that cannot be, in your opinion, captured by the vector DB. I probably didn't get your point.

Disclaimer: I work for Qdrant, and we believe a database should be just a database. I remember attempting to move logic to the database layer and coupling neural encoders into the vector database sounds the same.

[isoprophlex]

> 1. To use LLMs effectively, you often need to generate and store more than 1 vector per document.

Could you elaborate this point for me? What would cause the 1 document -> ~ 100 vectors blowup; do you store vector embeddings for sections of the document, or use multiple models to create several types of vectors?

[Dachande663]

If you look at something like LangChain[0], it supports/recommends splitting larger documents into smaller chunks. In this way, when doing something like semantic search you can get the specific paragraph/section that holds the closest relevance, rather than having to read the entire document again (think of a 100 page PDF).

https://python.langchain.com/en/latest/modules/indexes/text_...

[jeadie]

This is generally very context/use case specific. In general, if a document is a `Dict[str, Any]`, then you either have to have one (or multiple) vector(s) per field, unless you want to combine vectors across fields (it's not self-evident how you'd best do that). In saying that, specific reason's to do this (or why I've done it in the past).

1. Chunking long text fields in documents so as to get a better semantic vector for them (also you can only fit so much into an LLM). 2. Differently to 1. chunking long text fields (or even chunking images, audio, etc), is one way to perform highlighting. It helps to answer the question, for example, for a given document what about it was the reason it was returned? You can then point to the area in the image/text/audio that was most relevant. 3. You may want to run different LLMs on different fields (perhaps a separate multi-modal LLM vs a standard text LLM), or like another comment said have different transforms/representations of the same field.

Perhaps 100 vectors is non-standard, but definitely not unseen.

[jkb79]

Only Vespa allows you to index multiple vectors per schema field, avoiding duplicating all the meta data of the document into the "chunk", and avoids maintaining the document to chunk fan-out. See https://blog.vespa.ai/semantic-search-with-multi-vector-inde...

[Sai_]

I’m not a data scientist but I think I know why one document could lead to many vectors.

(Happy to be corrected and/or schooled.)

A vector is a list of numbers each of which represents weight accorded to a certain word along a certain dimension.

Let’s take an example.

Is an “apple” a “positive” or a “negative” thing? Most people would associate positivity with apples. So, for the general population, the vector for “apple” along the 0-1 continuum where 0 represents negative sentiment and 1 represents positive sentiment would be something like [0.8].

Let’s add one more dimension. Is an apple associated with computers (1) or not (0)? For the majority of the world where Windows has a massive market share, “apple” would recall a fruit, not a sleek laptop. Therefore, the vector for apple along the computer/non-computer dimension is probably [0.3].

Taking this together, apple = [0.8, 0.3] where positionally, 0.8 is the value for positive/negative sentiment while 0.3 is computer/non-computer.

Agree?

(Hoping you do)

But that [0.8, 0.3] vector is for the general population.

Would a bible literalist who publishes blogs on bible stories feel the same way?

For someone like that, the notion of the original sin could taint their sentiments towards the apple. So they might weight an apple at 0.2 on the positive/negative line. Since they’re bloggers, it’s more likely they associate apple with computers so they might call it 0.5. Therefore, their apple vector is [0.2, 0.5].

Extend this to more content and you’ll see why there are more than one vector.

At least that’s how I understood it. Happy to be corrected and/or schooled.

[ruslandanilin]

In my opinion, you could represent "apple" as a vector, for example, [0.99, 0.3, 0.7] in relation to [fruits, computers, religion]. Then, you can create different user vectors that describe the interests of various groups. For instance, the general population might have a vector like [0.8, 0.2, 0.1], geeks as [0.6, 0.95, 0.05], and religious people as [0.7, 0.1, 0.95].

By creating these user vectors, you can compare them with the "apple" vector and find the best match using ANN. This approach allows you to determine which group is most interested in a given context or aspect of the word "apple." The ANN will help you identify similarities or patterns in the user vectors and the "apple" vector to find the most relevant matches.

Thank you

[Sai_]

I don’t know what ANN is but your comment raises two questions in my mind -

1. Where did your first vector of [0.99, 0.3, 0.7] come from? You later present the concept of user vectors which are vectors for different cohorts of users but don’t name the first vector as a user vector.

2. I feel my example of vectors for “general population users” and “bible literalist blogger” user aligns with your “user vector” concept.

[thanatropism]

Modern text embeddings are not word-based like that.

[Sai_]

If my understanding and explanation are directionally correct, I’m happy. I’ll be the first one to admit I’m not a data scientist.

Do you have a good example of how an actual data scientist would present the idea of vectors as applied to sentences/documents to a layperson?

[teaearlgraycold]

Storing one document as one embedding is like making a movie poster the average of all frames in the film.

[jkb79]

That is a very good analogy!

[teaearlgraycold]

:D Thanks!

[amitport]

One thing others didn't mention is that "document" is a general term but in some cases (e.g., question answering) the typical document can be a very short paragraph and take much less memory than the vector. Also note that with some ML architecture the vector is very large (e.g., an entire very layer output)

[ethanahte]

Hi, author here.

1. You make a great point about longer documents requiring multiple vectors which I should've mentioned in the post. Depending on your use case, this can certainly explode your dataset size! 2. Good to know about the pgvector limitations -- I haven't used it yet. 3. I guess "index" would be the more database-y term. That said, one thing I'll call out is that you have to re-index if you ever change your embedding model, and indexing can be slow. It took me ~20-30 minutes to index the 10 million embeddings in my benchmark.

[kordlessagain]

I'm interested if anyone has some hard data on the "best" size of the document "fragments" that are used for embedding into a dense vector.

Obviously, embedding single words probably aren't particularly useful for reassembling portions of a document for submission to an LLM in the prompt. I'm currently pondering on what size of string is best for embedding, and considering a variable size might be one option.

Testing with strings around 512 characters seem to do pretty well, but it may be storing multiple lengths of similar runs in the document might be a better way to do it.

[rahimnathwani]

This will depend on the specific model you're using, because:

- if a model has been trained on shorter paragraphs, it will likely do better on those than on longer ones, and vice versa

- each model has some maximum input length (e.g. 512 tokens, or about 350 words), and might silently discard words when it's given a longer chunk

I don't know whether or not processing multiple lengths is worthwhile, but you probably want to have some overlap when you turn your docs into chunks.

Maybe take a look at Langchain or LlamaGPT: someone has probably come up with sensible defaults for overlap and chunk size.

If you want to do embeddings locally, check out sentence-transformers/all-MiniLM-L6-v2

[rahimnathwani]

On your last point: I guess recalculating 10 million embeddings takes much longer than the 20-30 mins to re-index?

Or perhaps we care because calculating the embeddings can be done in parallel with no limit, but the indexing is somehow constrained?

[ethanahte]

Yeah, depending on the model, calculating the 10 million embeddings could take longer sequentially, but, as you mention, it's also an embarrassingly parallel operation. I don't think that indexing can be performed in parallel, but I may be wrong on that one.

[heyzk]

Would love to hear more about (1), are you vectorizing different types of transforms on the original document?

[jn2clark]

I think you can and it has some benefits. One interesting thing that can help is to store representations from transformations over the document and then "fuse" the vectors (i.e. average them) at indexing time. You are effectively able to do run-time augmentation but without any extra inference overhead at query time and without increased memory. An easy way to think of this is for similarity measures that are linear (i.e. dot product) you are now scoring the document over a weighted sum of the transformations of the document. Test-time augmentation is a very well known method in ML generally for improving performance and is applicable here. You can do the same for queries as well - akin to query expansion.