Hacker News new | ask | show | jobs
by gk1 1579 days ago
We've had people kicking down our doors (figuratively) for the filtering feature. I have no doubt ES is getting the same feedback and is working on it.

> I wonder if there actually some fundamental reason they won't be able to do so.

This is where it gets interesting. I don't think there's anything a company with ES's resources can't do if they really want to. However...

ES's vector search is based on Lucene 9.0, which in turn uses the HNSW vector index. HNSW is shaky when it comes to CRUD updates (namely deletions), and flat out does not support metadata filtering: https://issues.apache.org/jira/browse/LUCENE-10040

Even with ES resources, it's a fundamentally hard problem. In the best case it's going to take a while to implement.

1 comments

If you control the HNSW implementation, it can definitely do pre-filtering. Vespa does it, and you can modify open source HNSW libs easily. I added pre-filtering support to an internal fork of HNSWLIB last week, for example…