Hacker News new | ask | show | jobs
by itake 1045 days ago
imho, vectorDbs need to scale horizontally. Simply running on a single host doesn't cut it anymore.
1 comments

You are right. We designed our storage in a segment-based way, with configurable segment size, so it can horizontally scale in the future cross multiple workers in one machine, and cross multiple machine cluster. And the search will become a two-stage search: find top K in each segment, then a global merger (can also be horizontally scaled) to merge the results from all segments