[vacuity]

For the syscall raw ASM/libc debate, would it be possible to provide an interface that just does syscalls and separate that from the rest of libc? It would be more inconvenient for people using ASM, but they wouldn't have to conform to libc. I imagine it's a breaking change for everyone, so consider this in a hypothetical OS.

[tsimionescu]

It might be possible, but I don't think you'd gain much. Even today, you can dynamically link to libc but only use its syscall interfaces, not anything else from it, not even malloc(). I think most runtimes for GC languages work like this.

However, this still means that your process will be affected by any memory correctness issues in "libsyscall", in addition to the issues in the kernel itself. Plus, the maintainers of libsyscall would have to write it in a bizarre dialect of C that doesn't use any stdlib functions, which might price even more error prone than standard C.

It's perhaps important to note here that the parts of libc that implement syscalls in OpenBSD are not simple syscall-to-C wrappers, they can have quite a bit of code occasionally. And Windows' runtime library is even more complex than that. That's their whole point - they can keep a backwards compatible system interface in spite of significant changes at the syscall layer, probably by doing lots of small pieces of work in userspace to bridge the gap.

[vacuity]

I don't think there's any way to mitigate memory safety issues in the syscall wrappers or the kernel; if the very overseer that is depended upon to enforce some degree of security isn't secure, then it can't be relied upon.

I was moreso thinking that "libsyscall" would be like libc, so that people can use it as a stable interface as in Windows or OpenBSD. If you were to use both libsyscall and libc, it wouldn't be meaningfully different from linking to all of libc today. It gives somewhat more separation to treat syscalls independently of libc, that's all.