[protosam]

Depending on what you’re using these tools for. If you want a locking manager and some meta data storage to help your distributed system maintain state, etcd is better for the job than rqlite for that. It’s a better zookeeper. With etcd you can hold a lock and defer unlocking if the connection is disrupted. Rqlite is not a good option for this.

[otoolep]

Agreed, in the sense that while rqlite has a lot in common with etcd (and Consul too -- Consul and rqlite share the same Raft implementation[1]) rqlite's primary use case is not about making it easy to build other distributed systems on top of it.

[1] https://github.com/hashicorp/raft

[protosam]

Every time I've looked at rqlite, it just falls short features-wise in what I would want to do with it. A single raft group does not scale horizontally, so to me rqlite is a toy rather than a tool worth using (because someone might mistake the toy as production grade software).

[otoolep]

rqlite creator here.

That's clearly a mistaken attitude because both Consul and etcd also use a single "Raft group" and they are production-grade software.

Ruling out a piece of software simply because it doesn't "scale horizontally" (and only writes don't scale horizontally in practice) is a naive attitude.

[protosam]

The qualifier here is for /my/ use cases. However I couldn't recommend rqlite over better options at the level of scale that it can fill.

One of the problems is if you're working with developers, the log replication contents is the queries, instead of the sqlite WAL like in dqlite. I know this is a work around to integrate mattn/sqlite3, but it's untenable in enterprise applications where developers are going to just think "oh, I can do sqlite stuff!". This is a footgun that someone will inevitably trigger at some point if rqlite is in their infrastructure for anything substantial. In enterprise, it's plainly untenable.

Another issue is if I want to architect a system around rqlite, it wont be "consistent" with rqlite alone. The client must operate the transaction and get feedback from the system, which you can not do with an HTTP API the way you've implemented it. There was a post today where you can observe that with the jetcd library against etcd. Furthermore to this point, you can't even design a consistent system around rqlite alone because you can't use it as a locking service. If I want locks, I end up deploying etcd, consul, or zookeeper anyways.

If I had to choose a distributed database with schema support right now for a small scale operation, it would probably be yugabyte or cockroachdb. They're simply better at doing what rqlite is trying to do.

At the end of the day, the type of people needing to do data replication also need to distribute their data. They need a more robust design and better safety guarantees than rqlite can offer today. This is literally the reason one of my own projects has been in the prototyping stage for nearly 10 years now. If building a reliable database was as easy as integrating sqlite with a raft library, I would have shipped nearly 10 years ago. Unfortunately, I'm still testing non-conventional implementations to guarantee safety before I go sharing something that people are going to put their valuable data into.

To simply say I'm "ruling out a piece of software because it doesn't scale horizontally" is incorrect. The software lacks designs and features required for the audience you probably want to use it.

Hopefully you find my thoughts helpful in understanding where I'm coming from with the context I've shared.

[otoolep]

Wow, a lot there. Thanks for your comments.

>One of the problems is if you're working with developers, the log replication contents is the queries, instead of the sqlite WAL like in dqlite.

I think you mean rqlite does "statement-based replication"? Yes, that is correct, it has its drawbacks, and is clearly called out in the docs[1].

>Another issue is if I want to architect a system around rqlite, it wont be "consistent" with rqlite alone. The client must operate the transaction and get feedback from the system, which you can not do with an HTTP API the way you've implemented it.

I don't understand this statement. rqlite docs are quite clear about the types of transactions it supports. It doesn't support traditional transactions because of the nature of the HTTP API (though that could be addressed).

>Furthermore to this point, you can't even design a consistent system around rqlite alone because you can't use it as a locking service. If I want locks, I end up deploying etcd, consul, or zookeeper anyways.

rqlite is not about allowing developers build consistent systems on top of it. That's not its use case. It's highly-available, fault-tolerant store, the aims for ease-of-use and ease-of-operation -- and aims to do what it does do very well.

>If I had to choose a distributed database with schema support right now for a small scale operation, it would probably be yugabyte or cockroachdb. They're simply better at doing what rqlite is trying to do.

https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some...

Of course, you should always pick the database that meets your needs.

>If building a reliable database was as easy as integrating sqlite with a raft library, I would have shipped nearly 10 years ago.

Who said it was easy? It's taken almost 10 years of programming to get to the level of maturity it's at today.

>They need a more robust design and better safety guarantees than rqlite can offer today.

That is an assertion without any evidence. What are the safety issues with rqlite within the context of its design goals and scope? I would very much like to know so I can address them. Quality is very important to me.

[1] https://rqlite.io/docs/api/non-deterministic/

[protosam]

> That is an assertion without any evidence.

This seems like a lack of knowledge issue. The problems with rqlite are inherit in it's design as I've already articulated. You can literally start reading jepsen analyses right now and understand it if you don't already: https://jepsen.io/analyses