Hacker News new | ask | show | jobs
by sciurus 3030 days ago
"making a F/OSS competitor"

So, Solr? Good luck getting SolrCloud set up on Kubernetes. ;-)

More seriously though, my answer to "has anyone actually tried to get ElasticSearch up and running lately?" is yes. I just worked on spinning up a cluster (using docker) at my current job. At my last two jobs I also managed ElasticSearch (without docker). There are plenty of gotchas with ElasticSearch, but I've never found the initial setup to be a challenge. To be fair, I've never touched X-Pack.
1 comments
hardwaresofton 3030 days ago
Call be insane +/- naive, but I was actually thinking of "just" gossiped/quorumed SQlite+FTS5.

In the end I got elastic search running, but it wouldn't connect to Kibana properly. I exaggerated too much -- much of my frustration was with ES not working properly with Kibana. I kept notes on what went wrong/what I was struggling with but I don't even want to look at them now, they'll be in a blog post someday

link
pas 3029 days ago
You'd need to handle concurrent writes, so something like a WAL, so why not build on RocksDB?

And okay, quorum, and sure there are a lot of Raft libs out there, but it's a bit harder than "new Cluster(Consistency.QUORUM)" :)

link
hardwaresofton 3028 days ago
The thing is, I don't want to build search myself -- SQLite has a WAL (of course), runs in memory if you want (of course RocksDB has less holding it back from utilizing memory even more efficiently than SQLite could), and most of the creature comforts of SQLite, and I can lean on SQlite FTS search.

All I have to get right is the quorum (I'm actually thinking optimistic gossip with something like swim over a quorum with paxos/raft), and the sharding, and replication -- and that stuff has been worked through by people much smarter than me already.

The formula I think will work is basically SQLite + SWIM/Raft + consistent hashing algo + optimistic replication + optimistic rebalancing. Just about 100% of the things on that list I don't have to think too hard to implement, and should be performant in the happy case (where n/2 nodes are up and healthy and relatively performant)

Recently saw a talk from fosdem 2018 (https://fosdem.org/2018/schedule/event/datastore/) about a project called Timbala, learned a bit from it (for example, assumed everyone was just using paxos/raft but SWIM evidently is used by consul)

link