Hacker News new | ask | show | jobs
by sidi 1137 days ago
There currently isn't a way to filter docs alongside a KNN query, and the dimension support is limited to 1024 (a Lucene limitation) and OpenAI embeddings are 1536 dimensions - also indexing performance is not comparable. Wishing this changes, as they're a good stack for the reasons you state
1 comments
softwaredoug 1137 days ago
True though I do think 2k dims is coming it 8.8
link
peterstjohn 1136 days ago
Are they forking Lucene or somehow getting the Lucene devs to increase that limit? Because this PR has been open for over a year now: https://github.com/apache/lucene/issues/11507
link
softwaredoug 1136 days ago
No - they just did something in Elasticsearch to make their own FieldType https://github.com/elastic/elasticsearch/pull/95257
link