[kefka]

Nope. Eventual Consistency seems to be the best policy.

The constant "C" isn't changing (or if it is, it's not much). that means there's a definite amount of time for a DB to be consistent to cover transit around the world. When the 3 DB machines are in the same rack, sure, we can approach 11.8"/nanosecond latencies, but the latencies are still there.

The larger and more global (and soon, interstellar), the more these C related lags become. And it is glaringly obvious to me, that "eventual consistency" is the big solution here.

The only case where it doesn't seem to be good, is transactional stuff like banking and investments. In those cases, seem to require CP and hardware like atomic clocks to verify every copy of the DB is on the same page.. Or perhaps an internal blockchain would be more appropriate, since it enforces consensus. But I digress on that one.

[freneticfox]

To go a little further, I suspect with CP cases like banks the real underlying problem is more fundamental. You can't realistically, under the laws of physics, have a hard notion of transactional ordering (did the account's money come in before it went back out?) without pinning down the concept of an account to a location. At least, not efficiently or quickly.

In other words, eventual consistency in the face of asynchronous remote actors never makes sense when your requirements dictate hard, consistent transactional ordering. You have to think of it as "the transactions happen in the order they arrive at the account's virtual location in New York". To think of them as globally-distributed in nature is always going to cause logical problems at some level. If your database was eventually-consistent, you'd have to build in some sort of after-the-fact safety checks that have the ability to abort the outer transaction, at which point you've wasted a lot of effort patching over the wrong model.

So either you have a need for strong consistency guarantees (order really matters), in which case you have to pin the transactions' locality down (where do they meet up at for their efficient strict ordering?) and CP is your model, or you don't (simpler things like social network updates) and you're better off with AP and eventual consistency to scale things out easier and make it faster for everyone, and really who cares if once in a great long while a user-visible race happens and some people see a couple of posts in a different order than someone else does for a few minutes until things snap back into sync?

[pjc50]

> You can't realistically, under the laws of physics, have a hard notion of transactional ordering (did the account's money come in before it went back out?) without pinning down the concept of an account to a location

You've just made me realise that the universe itself is only eventually consistent - that's what all those weird quantum observer / wavefunction collapse events are.

[kefka]

Yep, that's what I was trying to get at, but failed in my explanation.

Everything's eventually consistent on an ever-present sphere around the incident expanding at the speed of light. No faster.

Even if the sun were to blink out of existence, we'd have 8 minutes that we wouldn't know. Gravity would still be there, holding Earth in place. The light would still warm us. Then 500 seconds later, darkness. We'd be flung out on a tangential course.

[jacobparker]

This isn't quite right. Even with SR and the speed of light its possible to build consistent systems and achieve consensus. Not-eventually-consistent doesn't mean instantaneous. The SoL just sets a lower bound on the speed of consensus.

It's important to not overstate the importance of that bound, though.

[kefka]

Sure. CA fulfills that requirement. Of course, you throw away any semblance of partition tolerance.

Of course, a single machine guarantees there can be no partitioning, and really easy to obtain consensus. It might not be terribly fault-tolerant, however.

[jacobparker]

CA is a not really a valid/possible thing in the context of CAP. The original phrasing of the theorem was poor and the "choose 2" myth persists. You can choose to (or accidentally) give up C or A but you don't get to choose to not have partitions. Not being partition tolerant doesn't really make sense (you're just broken?) if partitions are going to happen. A better phrasing of CAP is "in a network with partitions a distributed system cannot be both consistent and available." (note: this doesn't guarantee that you are one of C or A, you just can't be C and A.) You can see that definition used in formal treatments, e.g. Theorem 1 in https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.p...

(Briefly, note that the original article is critiquing that definition of availability in practice which is legitimate but not relevant to this sub-thread.)

What I'm saying is that EC is most definitely NOT a requirement of physics/the speed of light (what your original post claimed.) The speed of light only sets a (theoretical) limits on how fast you can implement a consistent system.

The original Paxos paper ("The part-time parliament") uses an analogy of a quorum of parliamentarians occasionally getting together in the the same building and agreeing on something. Of course it being the same building is arbitrary and doesn't actually matter, but it's easier to intuit that the speed of light isn't an insurmountable road-block at that scale.

[jacobparker]

Special relativity and the speed of light impose some fundamental lower bounds on the cost of consistency: https://en.wikipedia.org/wiki/Relativity_of_simultaneity . This theoretically manifests as lower bounds on the performance of consistency in distributed systems.

[dragonwriter]

> The larger and more global (and soon, interstellar)

Soon, interstellar? That's...optimistic, to say the least.

[kefka]

I don't think so. It's not hard to see the beginnings of a satellite communication platforms around the Moon and Mars. And once we start looking at the local solar system, we start dealing with C being a significant source of lag.

For example, communications to and from Mars can take upwards to 20 minutes, depending on where the Earth is in relation to Mars. If this doesn't call for eventual consistency, I'm not sure what would.

The more humans grow, C is a thing we'll have to take in consideration for our communications mediums. Planet-wise comms still feel instant, but that's still on our ball of rock.

[dragonwriter]

> I don't think so. It's not hard to see the beginnings of a satellite communication platforms around the Moon and Mars.

That's not even remotely close to interstellar.

> And once we start looking at the local solar system, we start dealing with C being a significant source of lag

Sure, interplanetary has substantial impact from light-speed delays.

I'm just saying that pointing to imminent interstellar scope is wildly improbable.

[resu_nimda]

You mean interplanetary. It will be a while until Google has nodes orbiting Alpha Centauri.

[toomuchtodo]

https://en.wikipedia.org/wiki/Interplanetary_Internet

https://www.wired.com/2013/05/vint-cerf-interplanetary-inter...

Even if you discount manned travel, we have an entire solar system full of humanities remote sending gear in flight, hence the need for an IP-ish store and forward network.