[platz]

Distributed systems are not like traditional programs, because there is not just one "edge of the system".

Every node becomes an "edge" in it's own right, and doesn't necessarily have global coherence with the rest of the system.

[louthy]

True, but if the sender wants the receiver to do something of value then it will need to meet a contract that the receiver enforces. That doesn't require a central repository of contracts, one node can diverge, but you must understand that parts of your network of services will start to fail. From that point of view it starts to look very much like the linker phase of a compilation, and that the types need to match up to the data structures being instantiated. It's just this 'linking' phase is in the programmers heads, and not particularly useful. A distributed system that can validate itself is a much more valuable concept.

[platz]

Absolutely, the rubber meets the road at some point, nodes must understand/assume "contracts" about the data they are working with.

There already are static typed actor systems ( e.g. Orleans) which work well, but my point is that I believe OTP is more flexible for better or worse. Whether that flexibility is worth it to you for what you get is another matter.

Also I'm not sure how to think about binary compatibility between upgrades in such a system

[louthy]

> There already are static typed actor systems (e.g. Orleans)

Yep, I develop one myself. And have gone to the extent of not allowing senders to even post a message if it's of the wrong type (processes in nodes publish the types they accept to a central store). I initially went along with the 'accept anything' approach (which Akka really majors on too), but found that for the large systems I was developing that it became a real headache to deal with.

> but my point is that I believe OTP is more flexible for better or worse. Whether that flexibility is worth it to you for what you get is another matter.

Yep, fair enough, if it works for you, who am I to complain? It's not worth it for me, because I feel quite strongly that the code I write should understand the types it's working with. It feels like this super-late binding can give false positives, appear to work, when in fact it's not. That scares the shit out of me when systems get large.

[platz]

I gotcha, I am not even an erlang programmer. My two main languages are C# and Haskell and in general I abhor dynamic types.

All I am trying to do here is enumerate the difficulties in trying to Type "OTP" in Erlang (its main selling point) , and am not commenting on all possible actor systems.

[louthy]

Understood :)

[brightball]

Think of it this way: you have a struct type with 4 attributes that you want to pass to another function.

Currently, that function declares it will match on the pattern of those 4 attributes rather than a static type. Now, you update the Node and modify the type on the sending Node to have 5 attributes.

With pattern matching on the 4, everything still works. With static types on the struct the contract is now out of sync.

[louthy]

I find this to be a really poor argument. Essentially you're lucky if your systems continue to work as others go off changing message formats without consideration for the code that will receive it?

On a suitably complex/large system this is a recipe for disaster. Things start to slowly rot. It is far better to maintain the old function, accepting the old struct, map it to the new struct and forward it on to the new function that accepts the new struct. Let the old one consume anything that's already queued, or being sent from other nodes that haven't yet been upgraded whilst the new one takes the new format.

[cousin_it]

I've worked with systems like that for years, and it's fine. We can have several large binaries with different release schedules, passing around a big struct with 50 fields and many nested structs, with different people making changes to different parts. And nothing breaks. New code accepts old structs, old code accepts new structs, no conversion code required.

To achieve that, we follow the design of Protocol Buffers:

1) Each field in each struct has both a name and a numeric id. Only ids are used for serialization, so field names can be changed at any time.

2) All fields are marked as optional or repeated, never required. Most code is written to handle missing fields gracefully.

3) Changing the type or id of an existing field is forbidden. (Note that changing the contents of a nested struct doesn't count as changing its type.)

4) Adding a new field is okay, as long as you use an id that was never used before. (Each struct definition has a comment indicating the next available id to use.)

5) Removing a field is okay if you've checked that no one is using it anymore.

6) As a small but intentional bonus, you can change an optional field to repeated while preserving binary compatibility.

In the end it works out. You can think of breakages that could theoretically happen, but they don't.

[louthy]

What you're describing sounds like a manually implemented type system.

> Each field in each struct has both a name and a numeric id. Only ids are used for serialization, so field names can be changed at any time.

Fair enough your field names can be renamed. But the 'contract' is field numbers, not names.

> All fields are marked as optional or repeated, never required. Most code is written to handle missing fields gracefully.

So if all fields are optional, and you provide no fields at all, what happens? I assume the process rejects it, because it's not of the correct type?

> Changing the type or id of an existing field is forbidden.

Forbidden by what?

> Adding a new field is okay, as long as you use an id that was never used before. (Each struct definition has a comment indicating the next available id to use.)

I can understand this being the least problematic change to a type. But it still leads to 'if x has y field' behaviour, as your code tries to manage the full range of possible message types it might receive.

> Removing a field is okay if you've checked that no one is using it anymore.

That sounds super fluffy.

> As a small but intentional bonus, you can change an optional field to repeated while preserving binary compatibility.

Sorry, I don't follow? This bit confuses me 'change an optional field to repeated'.

> In the end it works out. You can think of breakages that could theoretically happen, but they don't.

I can think of many:

* If picking of IDs is done by a human, at some point a human will make a mistake and re-use an existing one

* If 'Changing the type or id of an existing field is forbidden' is a human enforced constraint, then it will fail

* If you think a certain struct pattern can't happen any more (you think you've retired all nodes that send the old format), and then you deprecate the many matches that deal with legacy messaging, and then realise that actually there is an old node that does it after all.

* You may re-add a field to a type which was previously removed and cause unexpected behaviour in parts of the system that match on that old format

* Removing a field that you thought wasn't used any more but actually still is

By the way, I'm not suggesting it's not possible to develop robust systems without a static type system of some sort; but I do think the hoops you're jumping through in items 1-6 indicate the problems of not using static types. Each change in functionality could just use a new struct, with a new function, and the old function maps to the old struct to the new one. It captures precisely the change in logic in one place, has no runtime cost for nodes that are sending the new struct, and can't lead to the edge cases that I listed above.

[cousin_it]

It helps that we have one big source control repository for the whole company, which runs many custom hooks that can block changes from being committed. For example, if I try to add a field whose id is lower than the next available id specified in the .proto file, my change won't commit. Same if I try to remove a field that's still used by someone else and my change breaks their build.

More generally, I think RPC interfaces need to be forward-compatible by design. If you have two binaries that are released on different schedules, and the API between them is fully rigid, how do you ever change it? Version the whole API, for a change that adds one boolean feature flag to one struct somewhere? Write a converter for fifty existing fields every time you add a new field, leading to O(n^2) programmer work over time? Come on.

Even more generally, I think static types are a great idea, but they work best locally. Communication over longer distances (in space and time) requires a different set of tradeoffs. There's a reason why people design network protocols and file formats with open-ended forward compatibility in mind. RPCs are kind of a middle ground, and I've found that the tradeoffs in Protocol Buffers work pretty well. YMMV.

[di4na]

False. In erlang, your message passing is "at most once".

If you send a bad message, the receiver will crash or discard it and it is how it is intended to be.

Erlang embrace laws of maths and physics.