[danieljomphe]

Right. So the only way to make Peers resilient to network partitions is to install a middleman between them and the DB/Transactor. One whose responsibility is to ensure this Peer's app always has durable access to everything it's ever going to need to be able to read for its queries, and always has durable access to some local write log that doesn't exist in the current implementation.

Thus my question is: is introducing such a middleman in the system going to denaturate Datomic?

[weavejester]

I don't believe Datomic is designed to operate in a scenario where Peers don't have network connectivity. The local cache Peers keep is to cut down on network traffic and improve performance, not as a reliable "offline mode".

[danieljomphe]

Notwithstanding what it's initially designed for, I think it may be quite good at supporting an "offline mode" as long as:

1. the app developer can confidently predict which queries the app will need through its lifespan, and

2. the app developer is willing to program and configure a layer that can persist and make durable a cache that spans all the data needed to run those queries (thus, persisting locally what amounts to a dynamic shard of the DB), and

3. the app developer is willing to program a layer that can persist and make durable all writes intended for the Transactor, and synchronize those to the Transactor when the app recuperates from a network partition, and

3.1. the app developer is willing to plan-or resolve-potential conflicts in advance of-or when-eventual conflicts, thus he's willing to sacrifice global consistency in the event of a network partition, in order to obtain availability, and

4. the app developer is willing to plug into the query engine in such a way that queries will include the local write log when there's a network partition.

Now, solution-wise:

1. depends on the requirements but most small to medium apps can predict the queries they'll need;

2. seems to be quite easy for small to medium apps:

2.1 run all possible queries at regular times, and

2.2 use a durable key-value store to keep the db values;

3. (1) make sure you're subscribed to events on partition and recovery; (2) coordinate writes over the same key-value store, probably using Clojure's STM and/or Avout; (3) on network recovery, replay those writes not present in the central DB;

3.1 due to the immutable nature of things and total ordering of the DB transactions, I expect to see no issue regarding eventual consistency when write logs are replayed centrally after a local Peer recovers from a network partition;

4. considering how Datalog works and is integrated into the Peer, this seems like a piece of cake.

So isn't this quite feasible to support the highly distributed case for apps in which each local Peer represents its own logical, dynamic and relatively natural and autonomous shard of the database?

[nickik]

Seams to be true but the intressting part is that peers can be made parallel and if one datacenter explodes you can go to an other without losing information. The only "Single Point of Failure" is the transactor and only for reads.

[weavejester]

You mean only for writes :)