I think there is. Every OS that exists is a mess. Make an OS whose inner workings I can undestand (and therefore are very well documented) while allowing to make commonly used software, and I will worship you.
There's no need for a new operating system if your criteria is just "inner workings I can understand". OpenBSD is very thoughtfully implemented, its documentation is top notch, it's open source, and yet it is a general purpose operating system that runs real world applications. It isn't a "mess"; in terms of its design, I think OpenBSD is about as clean as a real-life operating system can possibly be, so much so that my college operating systems professor even chose it over Minix for teaching the course.
I still think there's room for improving operating system security models, but I don't think it's possible to improve over OpenBSD in this particular dimension.
I still think there's room for improving operating system security models, but I don't think it's possible to improve over OpenBSD in this particular dimension.