I understand the need for writing this as an SDK over existing languages for adoption reasons, but in your opinion would a programming language purposely built for such a paradigm make more sense?
(Disclaimer: I work at Restate on SDKs) This is a very interesting point. I did some investigation myself, and so far I'm torn apart on whether a novel language would really make such a big difference for durable execution engines like Restate.
Let me elaborate it: first of all, what would be the killer feature that justifies creating a whole new PL for durable execution? From what I can tell, the thing that IMO can really make a difference would be the ability to completely hide durable execution from the user, by being able to take snapshots of the execution at any point in time and then record those in the engine transparently. Now let's say such language exists, and it can also take those snapshots reasonably fast, it is still quite a problem to establish where it's logically safe to take a snapshot, and when the execution cannot continue because you need to wait acknowledgment for stored results. Say for example you have the following code:
val resultA = callA()
val resultB = callB(resultA)
Both A and B do some non-deterministic operation, e.g. they perform HTTP calls to some other systems. Now let's say that when callB() completed, but before you got the HTTP response, your code for whatever reason crashes. If you didn't took any snapshot between callA() and callB(), you will completely lose forever the fact that B was invoked with resultA, and the next time you re-execute A, it might generate a result that is different from the one that was generated the first time. Due to this problem, you would still need to somehow manually define some "safepoints" where it's safe to take those snapshots. Meaning that we can't really hide the durable execution from the user, as you would still need some statement like "snapshot_here" to tell the engine where it's safe to snapshot or not.
In our SDKs we effectively implement that, by taking the safe approach of always waiting for storage acknowledgement when you execute two consecutive ctx.run().
Oh wow, thanks for the depth in your reply. well I don't know anything about programming languages but something just made me ask this question out of curiosity. I may have to play around with restate a bit
Super interesting question! If we were inventing modern tech from scratch, I think there's space for this, definitely. Our goal though is that people can use their primitives in the systems they have already, which means Java, Go, Python, TS support are all table stakes
Let me elaborate it: first of all, what would be the killer feature that justifies creating a whole new PL for durable execution? From what I can tell, the thing that IMO can really make a difference would be the ability to completely hide durable execution from the user, by being able to take snapshots of the execution at any point in time and then record those in the engine transparently. Now let's say such language exists, and it can also take those snapshots reasonably fast, it is still quite a problem to establish where it's logically safe to take a snapshot, and when the execution cannot continue because you need to wait acknowledgment for stored results. Say for example you have the following code:
val resultA = callA() val resultB = callB(resultA)
Both A and B do some non-deterministic operation, e.g. they perform HTTP calls to some other systems. Now let's say that when callB() completed, but before you got the HTTP response, your code for whatever reason crashes. If you didn't took any snapshot between callA() and callB(), you will completely lose forever the fact that B was invoked with resultA, and the next time you re-execute A, it might generate a result that is different from the one that was generated the first time. Due to this problem, you would still need to somehow manually define some "safepoints" where it's safe to take those snapshots. Meaning that we can't really hide the durable execution from the user, as you would still need some statement like "snapshot_here" to tell the engine where it's safe to snapshot or not.
In our SDKs we effectively implement that, by taking the safe approach of always waiting for storage acknowledgement when you execute two consecutive ctx.run().
But happy to be proven wrong!