|
|
|
|
|
|
by lmeyerov
1578 days ago
|
|
|
Interesting, thank you! Why does 100M vectors not work in ES? - Is this a configuration issue -- common for ES users -- or something fundamental? - It sounds like latency is the main thing. Any numbers intuition here, and any other dimensions of concern? AFAICT ES is using the same OSS vector libraries as pinecone, weaviate, etc. ES in general is used for > 100M documents, e.g., logging, so this is surprising. We are seeing growing interest by our ES/Splunk users in combining our viz tech with vector indexes, so I've been wondering about these, thanks! We currently go out-of-band at the compute tier or dump in our own indexes, but are thinking through managed flows, where fundamental limits gets interesting. |
|
|
I think that it's important to mention Elasticsearch version in this context :=)
Full disclosure, I work on Vespa.ai, but I have a pretty good understanding of how NN and ANN works in ES.
Elasticsearch 7x only had support for brute-force exact nearest neighbor search (1) which you can surely run with 100M documents, you just need to have a lot of nodes to bring latency down to service latency SLA. Typically, a single threaded/single index segment query with 1M documents with 128 dims using float is 300ms with Elasticsearch 7x (3). Two shards/segments instead would bring it down to 150 ms and so on. But 100M becomes costly, but perfectly doable.
With Elasticsearch 8.0, there is support for doing approximate nearest neighbor search (2) using the HNSW support from Apache Lucene 9. This would bring latency down to low single digit ms (single threaded, single shard) with 1M documents. The biggest disappointment was probably that they don't integrate the approximate nearest neighbor search with regular query filters/terms. This will likely be coming in Elasticsearch 8.1 or so, once it's implemented in Apache Lucene 9.x.
(1) https://www.elastic.co/blog/text-similarity-search-with-vect... (2) https://www.elastic.co/blog/introducing-approximate-nearest-... (3) https://github.com/jobergum/dense-vector-ranking-performance