|
|
|
|
|
|
by maxogden
3166 days ago
|
|
|
The real-time syncing immutable, append-only log that hypercore provides can also be accessed randomly, allowing for lots of cool parallel/distributed log processing architectures (Kappa architecture), similar to Kafka (which is also based on immutable, append-only logs). We have just focused on syncing a filesystem at first because we had a strong use case there, but you could totally build a CRDT on top of an append only log. |
|
|
Is random access something you guys have added since then? Or, could you clarify how it reduces the overhead on apps that would have shared mutable state (even if it is composed of immutable streams)? Don't you still have to back-scan through the log to recompose the current view? Which then wouldn't scale for read performance. Or is there an alternative approach?
This was one of the main things I was discussing with Dominic Tarr about Scuttlebutt back then. We had done a lot of event-sourcing (immutable logs) at my previous startup, but had problems with compaction or generating views/state of those streams. Which is what lead me to the CRDT approach as the base, not the thing on top. I know Tim Caswell/Creationix is using DAT for one of his products he is building with @coolaj86 .
But that was 4 years ago now, that me and Dominic were talking about those problems? Does random access solve that? Would love to read about it, shoot me some links! Also, Dominic and I are starting to have Web Crypto meetings with some folks at MIT (anvil/trust, ex-riak/basho) and they are in town (the Bay Area) this week. If you are around, we should all get together again to discuss identity, security, crypto, and P2P!