Hacker News new | ask | show | jobs
by marknadal 2355 days ago
w00h00, this is my area of expertise.

After 8 years of working on this, I have changed my thoughts:

- The correct algorithm is not always the correct user experience.

- End-to-end encryption is too important to not have.

- Offline support is great, but it behaving consistently is more important than it behaving "intently".

- Biggest pain points can most easily be solved at the editing layer, not data layer.

As a result, OT gets thrown out the door immediately.

I built the most popular CRDT powered database on the market (https://github.com/amark/gun, 11K+ stars, 11M+ monthly installs), but have some harsh words for CRDT obsessed people:

CRDTs are great, but if you create an infinitely complex one to handle "every word editing operation" it'll actually result in an inferior user experience.

As such, for example, GUN supports lock-free concurrent operations on any depth of data in a graph, but keeping text/strings behavior atomic is way more important than having built-in automatic string merges.

Why? It is much simpler to build a correct text/string CRDT on top of a predictable, stable, graph CRDT that is understandable.

Another example is, in our blog editing tools, despite having spent years researching & implementing (+ others in our community) precise character-by-character CRDT resolution schemes, I found I personally had a much better offline & local first user experience with a predictable per-paragraph sync scheme.

This is a stupid simple approach, but what it means is that I as a user, can easily predict whats going to happen, so if I'm offline I know I should just copy a new paragraph and fiddle with things there, cause it isn't gonna get "auto delete regrammar merged".

Generally speaking, me and colleagues don't "fight" over the same paragraph, we're usually concurrently writing different "sections" at the same time, it is pretty rare to grammar fix their edit same paragraph as they're typing it - that is also just kinda, rude looking.

Anyways, finally, is that the majority of text styling and DOM edge cases can be handled by having a deterministic canonical DOM hierarchy that always gets applied at the editing layer, BEFORE any CRDT operations even occur.

So for instance, re-arrange the DOM tree such that <i> is always inside <u> inside <b>, or something like that.

We built this into a library called normalize, and it instantly eliminated so much complexity, especially at the CRDT level. Ping me if you want a demo of this library.

Finally, for everyone else, we also built an INTERACTIVE CARTOON EXPLAINER of an extremely basic text CRDT to help others understand the more detailed concepts:

https://gun.eco/explainers/school/class.html

4 comments

User experience is definitely why we didn't go for an existing CRDT solution.

The model we've chosen is fairly agnostic to whether OT or CRDT is used to collaborate, as I'll expand on in the second post, so if we see an opportunity to switch to a CRDT-based solution we'll certainly jump on it!

You only need to normalise things like bold/italic/underline when using a browser DOM, every collaborative rich text model I've seen represents formatting as unordered attributes which is much easier to collaborate on.

You are a charming and energized speaker and perhaps that is why Tim Draper thinks you a good investment as with others he has made. Acronyms like 'PTSD' and 'PARTY' in the project is likely good at attracting people who think they sound fun and do not question many details.

It is difficult to get brief clear details about how gun.js works. There are distracting or simple documents, discussions about high level ideas and tangentially-related topics. In some cases 'worse is better' yes but it sets a tone that the project is hiding internal problems of implementation or can not easy explain its own design.

The transparency of gun.js enterprise organizations also raises questions. Are people on your team related to you? Do you pay yourself and team and open source contributors?

Thank you!

They also like the math & science behind it, some investors do DD, some don't.

You're tone is pretty assuming, smug, and insulting.

It is running in production with millions of people, yes there are some hiccups, but surviving & we're improving it to scale even more.

I rebuild GUN from scratch in 30min on stage, that is how little/simple GUN is:

https://gun.eco/docs/Porting-GUN

Do I pay myself? Lol. Yes. No, I do not pay OSS, but yes people contribute regardless, strangers help me, my kids help me, investors help me, family & friends help me.

With end-to-end encryption in GUN, what role does the server play? I assume the merging type activity happens in the clients?

BTW, the name makes it almost impossible to search for info about GUN.

Yes, merging is per peer, not server transformed.

"Servers" act as WebRTC bootstrapping signals, storage backups, relay/routing, etc., and you can have as many of them as you want (no need to be centralized).

True. `gunDB` tag should help a little.

Do you have any document that explain what resolution algorithm uses in what cases?

For example, one peer change a property value and the other peer deletes it.

Same algorithm for everything.

This cartoon explains it:

https://gun.eco/distributed/matters.html

It is a vector + timestamp + lexical sort.

A delete is changing a value to `null`, it would lose (I assume you are asking if/when these 2 changes happen at the exact same microsecond time, conflicting?) as is it has a lower lexical rank.

Can you refer me to a document where you explain the "lexical rank" that you are referring to?

The cartoon explanation do not tackle those terms.

JSON.stringify('a') < JSON.stringify('z')

https://en.wikipedia.org/wiki/Lexicographical_order

Cool! Make sense.

Let's say our state have an array like ['a','b'].

What about if Peer-A splice the first element (0) which results in ['b'] and Peer-B wants to mutate 'a' to 'A' which results in ['A','b'] and they both make the mutation at the same millisecond?

You have to choose which type of Array:

1. An atomic Array

2. Fixed index Array

3. Linked list

Now rather than not knowing what magic is going to happen, app developers know what to expect in advance.

So atomic would choose one or the other, not a combined result.

Fixed array 2nd item to 1st which nulls 2nd and makes 1st b, simultaneously also makes 1st A. Cause lower case b is lexically larger than A, you'd get [b, null].

Linked list theoretically could handle it as you suggest.