Hacker News new | ask | show | jobs
by hardwaresofton 3032 days ago
Call be insane +/- naive, but I was actually thinking of "just" gossiped/quorumed SQlite+FTS5.

In the end I got elastic search running, but it wouldn't connect to Kibana properly. I exaggerated too much -- much of my frustration was with ES not working properly with Kibana. I kept notes on what went wrong/what I was struggling with but I don't even want to look at them now, they'll be in a blog post someday
1 comments
pas 3031 days ago
You'd need to handle concurrent writes, so something like a WAL, so why not build on RocksDB?

And okay, quorum, and sure there are a lot of Raft libs out there, but it's a bit harder than "new Cluster(Consistency.QUORUM)" :)

link
hardwaresofton 3031 days ago
The thing is, I don't want to build search myself -- SQLite has a WAL (of course), runs in memory if you want (of course RocksDB has less holding it back from utilizing memory even more efficiently than SQLite could), and most of the creature comforts of SQLite, and I can lean on SQlite FTS search.

All I have to get right is the quorum (I'm actually thinking optimistic gossip with something like swim over a quorum with paxos/raft), and the sharding, and replication -- and that stuff has been worked through by people much smarter than me already.

The formula I think will work is basically SQLite + SWIM/Raft + consistent hashing algo + optimistic replication + optimistic rebalancing. Just about 100% of the things on that list I don't have to think too hard to implement, and should be performant in the happy case (where n/2 nodes are up and healthy and relatively performant)

Recently saw a talk from fosdem 2018 (https://fosdem.org/2018/schedule/event/datastore/) about a project called Timbala, learned a bit from it (for example, assumed everyone was just using paxos/raft but SWIM evidently is used by consul)

link