[ori_b]

> build it in the protocol!

You don't have access to the interprocess cache snooping in software. This is CPU interconnect internal shit, and you actually need access to the local memory bus for correctness. mmap in its fully glory is only really worth having if you can share pages from the buffer cache.

And even if you did, and you turn a ten nanosecond operation into a ten millisecond operation, counting the network packets you send (a factor of a million overhead), without the assumption that all the peers are reliable and never fail, the abstraction still breaks. And if you assume all your peers are reliable in a distributed system, you're wrong. Damned either way.

> I've never seriously looked at 9p, but the page you linked strongly suggests to me that it's more abstraction if anything

No, it's a single abstraction, instead of dozens that step on each other's toes.

> Great. I get it. 9p is a basic transport method that gives some introspection for free

What introspection? It's just namespaces and transparent passthrough. Unless you're talking about file names.

[jstimpfle]

Yes, I realized the CPU issue and already updated my comment. Technically we would need a way to catch the CAS operation and convert it into a network request - like for example segfaults can be handled and converted into a disk load.

And also we'd need to extend all the cache coherency stuff over the network.

> And if you assume all your peers are reliable in a distributed system, you're wrong. Damned either way.

Technically you have the reliability issues with all the components inside a single system, just as well. They are just more reliable. But I'm sure I have seen hard disks failing, etc.

--

Ok, let me think about that abstraction stuff. Thanks.

[ori_b]

I'd also argue that if you need to turn a cache snoop into a network round trip (or several?), your abstraction is just as broken as if it returned the wrong value; it's unusably slow :)

[jstimpfle]

Yes, that's why I'm still having trouble to understand the fuss about VFS abstraction as used in 9p etc ;-). I've always been glad to know when I was not on an NFS or sshfs mount (mostly for reasons that you can't design away, i.e. network reliability issues). So why bother abstracting out that knowledge even more?

[ori_b]

When you come at it from the perspective that remote resources are the norm, and you assemble a computer from resources scattered across the network, local access becomes weird special case. Generally, you're running a file server and terminal as separate nodes, with an auth server somewhere else.

And if you're actually using the knowledge that some files are local, you get bugs and assumptions creep in, and now your software stops working in the case where you're running on someone else's network.

It's about transparently providing the same interface to everyone, and making that interface simple enough that implementing it is easy enough that it's actually done, and the interface actually gets used.

Then, if you want to interpose, remap, analyze, manipulate, redirect, or sandbox it, you can do that without much trouble. The special cases are rare and can be reasoned about.

Reasoning about your system in full frees you to do a huge amount.