[duped]

In practice you run all your process-isolated code in one process as a daemon instead of spawning it per call.

I think you're on the right track in boiling down the problem to "minimize the memory shared between languages and restrict the interface to well defined semantics around serialized data" - which in modern parlance is called a "remote procedure call" protocol (examples are gRPC and JSON RPC).

It's interesting to think about how one could do an RPC call without the "remote" part - keep it all in the same process. What you could do is have a program with an API like this :

    int rpc_call (int fd); 

where `fd` is the file descriptor (could even be an anonymous mmap) containing the serialized function call, and the caller gets the return value by reading the result back.

One tricky bit is thread safety, so you'd need a thread local file for the RPC i/o.