[maxpert]

Yes explored that path too with CRDTs.

- DDL gets really tricky in these cases, that's why you see Corrosion has this weird file based system. - cr-sqlite ain't maintained anymore but I did some benchmarks and if I remember correctly it was as slow as 4x-8x depending upon type of your data & load. Storage bloats by 2x-3x, tombstones accumulate pretty fast as well.

I mean each mutation on every column looks something like:

table, pk, cid, val, col_version, db_version, site_id, cl, seq

Overall I dropped the idea after spending month or two on it.

[wgjordan]

Very helpful hearing about your own similar experiments with CRDTs. As a followup I'd be interested in more direct comparison between Marmot and Corrosion in terms of features/performance, since they both serve a similar use case and Corrosion seems to have worked through some of the CRDT issues you mentioned.

[maxpert]

Ok it's a very long discussion but I will try to keep it brief here (more than happy to chat on Marmot Discord if you wanna go deeper). Honestly I've not done head to head comparison, but if you are asking for guestimated comparison:

- Marmot can give you better easy DDL and better replication guarantees.

- You can control the guarantees around transactions. So if you're doing a quorum based transaction, you are guaranteed that quorum has written those set of rows before returning success. This takes care of those conflicting ID based rows getting overwritten that people would usually ignore. And you should be able to do transactions with proper begin and commit statements.

- Disk write amplification is way lower than what you would see in CRDT. This should usually mean that on a commodity hardware you should see better write throughput. As I mentioned on my local benchmarks I'm getting close to 6K insert ops. This was with a cluster of three nodes. So you can effectively multiply it by three and that is like 18k operations per second. I did not set up a full cluster to actually benchmark these. That requires investing more money and time. And I would be honestly frugal over here since I am spending all my $$$ on my AI bill.

- Reads as you can see, you can read directly from the SQLite database. So you are only bottlenecked by your disk speed. There are no fancy mergers that happen on CRDT level in the middle. It's written once and you're ready to read.

- The hardest part in my opinion that I faced was the auto increment IDs. It is a sad reality but turns out 99% of small to mid-size companies, are using the auto increment for IDs. In all CRDTs, in case of conflict, the LWW (based on one ID or another) happens, and I can guarantee you at some point in time without coordination, if nodes are just emitting those regular incrementing IDs, THEY WILL OVERWRITE each other. That was the exact problem in the first version of Marmot.

- SQLite is single writer database. cr-sqlite writes these delta CRDT rows in a table as well, under high write load you are putting too much pressure on WAL, how do I know? I did this in Marmot v0.x and even v2 started with that and eventually I decided to write logs in a SQLite database as well. Turns out at a high throughput even writing or dumping those logs that change logs that I'm gonna discard away is a bad idea. I eventually move to PebbleDB, with mimalloc based unmanaged memory allocator for serialization/deserialization (yes even that caused slowdowns due to GC). It doesn't stop here each row in CRDT entry is for one every column of table (changed column) + it has index for faster lookup. So there that will bog it down further on many many rows. For context I have tested Marmot on gigs of data not megs.

I do have couple of ideas on how I can really exploit the CRDT stuff, but I don't think I need it right now. I think most of stuff can be taken care of if I can build and MVCC layer on top.

[dotancohen]

  > In all CRDTs, in case of conflict, the LWW (based on one ID or another) happens

In my syncing voicenotes application I've begun using UUIDv7 for primary keys, it's working out very well. The database is SQLite. INSERTs are a tad slow, but it has not been a problem in practice. Perhaps I've not deployed and tested enough, but I really feel this was a safe choice.

UUIDv7 does leak create time information, but it INSERTs faster than UUIDv4 because all the INSERTs happen at (or near) the end of the tree on sync.