My company is using vector search with Elasticsearch. It’s working well so far. IMO Elastic will eat most vector-first/only products because of its strength at full-text search, plus all the other stuff it does.
I tend to agree - search, and particularly search-for-humans, is really a team sport - meaning, very rarely do you have a single search algo operating in isolation. You have multiple passes, you filter results through business logic.
Having said that, I think pgvector has a chance for less scale-intense needs - embedding as a column in your existing DB and a join away from your other models is where you want search.
I don’t get why you’d want to bolt RBAC onto these new vector dbs, unless it’s because they’ve caused this problem in the first place…
They have beef with ES since they took the software, made a bunch of cash on it, then never contributed back. ES called them out and it started a feud.
I'd go on ES over Amazon-built software any day. I worked on RDS and I've used RDS at several companies, it's a mess.
Longer story:
One day one of our table went missing on Aurora, we couldn't figure out why, it was in the schema, etc. Devops panicked and restarted the instance, and then another table was missing. We ended up creating 10 empty tables and restarted it until it hit one of those.
We contacted RDS support after that, and the conclusion of their 3 month investigation is: "Yeah, it's not supposed to do that."
There's some really smart people working at Amazon, unfortunately the incentives is to push new stuff out and get promoted ASAP. If you can do that better than others and before your house of cards falls, you're safe. If the house of card crumbles after you're gone, it's their problem.
>Longer story: One day one of our table went missing on Aurora, we couldn't figure out why, it was in the schema, etc. Devops panicked and restarted the instance, and then another table was missing. We ended up creating 10 empty tables and restarted it until it hit one of those.
Are there any report this? How come this is the first time I heard of this? How can companies trust this kind of managed DB services?
We worked with dedicated support on this, but I don't think they had enough knowledge to dig deep into it and just gave up. There is a huge backlog of critical issues at most AWS services. It looks great from the outside in, but the sausage making process is extremely messy.
Amazon forked ElasticSearch into OpenSearch. When deciding which platform to go with (we are an AWS customer) I decided to stick with the company whose future depends on their search product (Elastic), not the one that could lose interest and walk away and suffer almost no consequences (AWS). If OpenSearch is still around in 5 years, and keeping pace with ElasticSearch, then maybe I'd consider it the next time I'm making this choice.
Also there's a lot more to ElasticSearch than full-text search (aggregations, lifecycle management, Kibana). Doesn't seem like Kendra is going to be a replacement for our use case.
Having said that, I think pgvector has a chance for less scale-intense needs - embedding as a column in your existing DB and a join away from your other models is where you want search.
I don’t get why you’d want to bolt RBAC onto these new vector dbs, unless it’s because they’ve caused this problem in the first place…